How to run DNN modules with CUDA backend support on Windows platform?
Overview In 2019, a commit was merged that added the option to use the CUDA backend for inference in OpenCV’s DNN module, resulting in faster inference on Nvidia GPUs. If you wish to use this CUDA backend with OpenCV for Unity, it is necessary to build and set up OpenCV as a dynamic library on your own. This article will provide guidance on the process. Steps Build the OpenCV library with the OPENCV_DNN_CUDA flag enabled. Please refer to the article below for instructions on how to build OpenCV with OPENCV_DNN_CUDA enabled. https://learnopencv.com/how-to-use-opencv-dnn-module-with-nvidia-gpu-on-windows/ https://www.jamesbowley.co.uk/qmd/opencv_cuda_python_windows.html https://medium.com/geekculture/setup-opencv-dnn-module-with-cuda-backend-support-for-windows-7f1856691da3 Build with OPENCV_EXTRA_MODULES_PATH and BUILD_SHARED_LIBS enabled. In addition, it is recommended that you download and build the same revision of the source as the OpenCV git link provided in the ReadMe.pdf. OPENCV_EXTRA_MODULES_PATH:PATH=C:/Users/xxxxx/opencv_contrib/modules BUILD_SHARED_LIBS:BOOL=ON Copy the built dll files to the Plugins folder. Replace the opencvforunity.dll file in the Extra package with opencvforunity.dll in the Plugins folder. To enable the CUDA backend, change the code as follows. net.setPreferableBackend( Dnn.DNN_BACKEND_CUDA); net.setPreferableTarget( Dnn.DNN_TARGET_CUDA); Result
How to combine ARExample and URP
The Universal Render Pipeline (URP) is a prebuilt Scriptable Render Pipeline, made by Unity. OpenCVforUnity itself works fine with any render pipeline, but the ARExample, which superimposes two camera images, does not render well in the URP project. This is because some components of the ARExample are configured for the built-in rendering pipeline. By setting these settings for the universal rendering pipeline, AR objects will render correctly. MainCamera ARCamera Cube MockARMarker ScreenShot
How to catch native OpenCV’s errors code (CVException handling)
In order to display the native opencv’s error code, please enclose the code in Utils.setDebugMode(true) and Utils.setDebugMode(false). Example Code: // // CVException handling example // // 32F, channels=1, 3×3 Mat m1 = new Mat (3, 3, CvType.CV_32FC1); m1.put (0, 0, 1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f, 7.0f, 8.0f, 9.0f); // 8U, channels=1, 3×3 Mat m2 = new Mat (3, 3, CvType.CV_8UC1); m2.put (0, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9); // dump Debug.Log (“”m1=”” + m1); Debug.Log (“”m1.dump()=”” + m1.dump ()); Debug.Log (“”m2=”” + m2); Debug.Log (“”m2.dump()=”” + m2.dump ()); #if UNITY_STANDALONE || UNITY_EDITOR // Publish CVException to Debug.LogError. Utils.setDebugMode (true, false); Mat m3 = m1 / m2; Utils.setDebugMode (false); // Throw CVException. Utils.setDebugMode (true, true); try { Mat m4 = m1 / m2; } catch (Exception e) { Debug.Log (“”CVException: “” + e); } Utils.setDebugMode (false); #else Debug.Log (“”The setDebugMode method is only supported on WIN, MAC and LINUX.””); #endif Execution Result: m1=Mat [ 3*3*CV_32FC1, isCont=True, isSubmat=False, nativeObj=0x820637680, dataAddr=0x820295296 ] m1.dump()=[1, 2, 3; 4, 5, 6; 7, 8, 9] m2=Mat [ 3*3*CV_8UC1, isCont=True, isSubmat=False, nativeObj=0x820637792, dataAddr=0x820619712 ] m2.dump()=[ 1, 2, 3; 4, 5, 6; 7, 8, 9] core::divide_12() : OpenCV(3.4.1-dev) C:\Users\xxxxx\Desktop\opencv\modules\core\src\arithm.cpp:683: error: (-5) When the input arrays in add/subtract/multiply/divide functions have different types, the output array type must be explicitly specified in function cv::arithm_op m3=Mat [ […]
Mat Basic Processing2
These codes are included in the OpenCVForUnity Example Unity scenes. (MatBasicProcessingExample) Shallow copy and deep copy Example Code: Execution Result: mat1=[1, 2, 3; 4, 5, 6; 7, 8, 9] m_shallow=[1, 2, 3; 4, 5, 6; 7, 8, 9] m_deep1=[1, 2, 3; 4, 5, 6; 7, 8, 9] m_deep2=[1, 2, 3; 4, 5, 6; 7, 8, 9] mat1=[100, 2, 3; 4, 5, 6; 7, 8, 9] m_shallow=[100, 2, 3; 4, 5, 6; 7, 8, 9] m_deep1=[1, 2, 3; 4, 5, 6; 7, 8, 9] m_deep2=[1, 2, 3; 4, 5, 6; 7, 8, 9] mat1.Equals(m_shallow)=True mat1.Equals(m_deep1)=False mat1.Equals(m_deep2)=False Merge Example Code: Execution Result: m_merged=[1, 1.1, 1.2, 2, 2.1, 2.2; 3, 3.1, 3.2, 4, 4.1, 4.2] MixChannels Example Code: Execution Result: m_mixed1=[1, 1.1, 2, 2.1; 3, 3.1, 4, 4.1] m_mixed2=[1.2, 1.1, 2.2, 2.1; 3.2, 3.1, 4.2, 4.1] Split Example Code: Execution Result: [1, 4, 7; 10, 13, 16] [2, 5, 8; 11, 14, 17] [3, 6, 9; 12, 15, 18] Reduce Example Code: Execution Result: m1=[1, 5, 3; 4, 2, 6; 7, 8, 9] v1(sum)=[12, 15, 18] v2(avg)=[4, 5, 6] v3(min)=[1, 2, 3] v4(max)=[7, 8, 9] m1=[1, 5, 3; 4, 2, 6; 7, 8, 9] v1(sum)=[9; 12; 24] v2(avg)=[3; 4; 8] v3(min)=[1; 2; 7] v4(max)=[5; 6; 9] RandShuffle Example Code: Execution Result: m1(original)=[1, 2, 3, 4, 5; 6, 7, 8, 9, 10; 11, 12, […]
Mat Basic Processing1
These codes are included in the OpenCVForUnity Example Unity scenes. (MatBasicProcessingExample) Initialization Example Code: Execution Result: mat1=[1, 2, 3; 4, 5, 6; 7, 8, 9] mat2=[0.8660254037844387, -0.4999999999999999; 0.4999999999999999, 0.8660254037844387] mat3=[1, 1, 1, 1, 1; 1, 1, 1, 1, 1; 1, 1, 1, 1, 1; 1, 1, 1, 1, 1; 1, 1, 1, 1, 1] mat4=[0, 0, 0, 0, 0; 0, 0, 0, 0, 0; 0, 0, 0, 0, 0; 0, 0, 0, 0, 0; 0, 0, 0, 0, 0] mat5=[1, 0, 0, 0, 0; 0, 1, 0, 0, 0; 0, 0, 1, 0, 0; 0, 0, 0, 1, 0; 0, 0, 0, 0, 1] mat6=[5, 5, 5; 5, 5, 5; 5, 5, 5] mat7=[246, 156; 192, 7; 165, 231] mat8=[124, 140; 125, 122; 131, 133] mat9.dims=4 mat9.rows=-1 //When the matrix is more than 2-dimensional, the returned size is (-1, -1). mat9.cols=-1 Multi Channel Example Code: Execution Result: mat1 dim:2 elemSize1:8 channels:1 mat2 dim:2 elemSize1:8 channels:10 mat3 dim:4 elemSize1:8 channels:1 Dump Example Code: Execution Result: mat1=Mat [ 3*3*CV_8UC1, isCont=True, isSubmat=False, nativeObj=0x793787296, dataAddr=0x794435328 ] mat1.dump()=[ 1, 1, 1; 1, 1, 1; 1, 1, 1] mat2=Mat [ 3*3*CV_8UC4, isCont=True, isSubmat=False, nativeObj=0x793789648, dataAddr=0x799875584 ] mat2.dump()=[ 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4; 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4; 1, 2, 3, 4, 1, […]
Way to translation of Mat class operators defined in C++
This is a list of implemented matrix operations that can be combined in arbitrary complex expressions (here A, B stand for matrices ( Mat ), s for a scalar ( Scalar ), alpha for a real-valued scalar ( double )): Note: In C++, the left-hand operand of compound assignment operators like “A += B” is reused, and operations such as “Core.add(A, B, A)” are performed internally. However, in C#, it is not possible to explicitly overload compound assignment operators. Instead, binary operator overloading is used implicitly, which results in a new Mat object being created and assigned to A each time an operator is used. This behavior leads to different memory management between C++ and C#. c++OpenCVForUnity(C#)Addition, subtraction, negation: A+B, A-B, A+s, A-s, s+A, s-A, -AA + BM1 + M2Core.add (M1, M2, M_dst)A – BM1 – M2Core.subtract (M1, M2, M_dst)A + sM1 + sCore.add (M1, s, M_dst)A – sM1 – sCore.subtract (M1, s, M_dst)-A-M1Core.multiply (M1, Scalar.all (-1), M_dst)Scaling: A*alpha A/alphaA * αM1 * 3Core.multiply (M1, Scalar.all (3), M_dst)A / αM1 / 3Core.divide (M1, Scalar.all (3), M_dst)Per-element multiplication and division: A.mul(B), A/B, alpha/AA.mul(B)M1.mul(M2)M1.mul (M2)A / BM1 / M2Core.divide (M1, M2, M_dst)α / A3 / M1Core.divide (new Mat (M1.size (), M1.type (), Scalar.all (3)), M1, M_dst)Matrix multiplication: A*BA * BM1 * M2Core.gemm (M1, M2, 1, new Mat (), 0, M_dst)Comparison: A cmpop B, A cmpop alpha, alpha cmpop A, where […]
How to extract the shape predictor from “mmod_dog_hipsterizer.dat”
First of all, please look at this article. Hipsterize Your Dog With Deep Learning (dlib.net)The “mmod_dog_hipsterizer.dat” cannot be used as a shape predictor data because it is distributed in a state in which combines multiple files. I will introduce here because I wrote the code for the use of the dog face shape predictor data in “DlibFaceLandmarkDetector”.This example scene was created based on the “DlibFaceLandmarkDetector/Examples/CatDetectionExample”.Replace the “CatDetectionExample.cs” that is attached to Quad to this script code.Set to the dog image to the texture2D inspector.The file that you downloaded and unzipped from here must be put to the “streamingAssets” folder.using UnityEngine; using System.Collections; using System.Collections.Generic; using System.IO; #if UNITY_5_3 || UNITY_5_3_OR_NEWER using UnityEngine.SceneManagement; #endif using DlibFaceLandmarkDetector; namespace DlibFaceLandmarkDetectorSample { /// <summary> /// Dog detection example. /// </summary> public class DogDetectionExample : MonoBehaviour { /// <summary> /// The texture2 d. /// </summary> public Texture2D texture2D; //https://github.com/davisking/dlib/raw/master/examples/faces/dogs.jpg private Texture2D m_tex; private int m_width, m_height; private Color[] m_linesColor = null; // Use this for initialization void Start () { gameObject.transform.localScale = new Vector3 (texture2D.width, texture2D.height, 1); Debug.Log (“Screen.width ” + Screen.width + ” Screen.height ” + Screen.height + ” Screen.orientation ” + Screen.orientation); float width = gameObject.transform.localScale.x; float height = gameObject.transform.localScale.y; float widthScale = (float)Screen.width / width; float heightScale = (float)Screen.height / height; if (widthScale < heightScale) { Camera.main.orthographicSize = (width * (float)Screen.height / (float)Screen.width) / […]