About OpenCV for Unity
Overview
https://assetstore.unity.com/packages/tools/integration/opencv-for-unity-21088
Support Platforms
OpenCV for Unity | |
|---|---|
| EditorPreview | ○ |
| Windows | ○ |
| Mac | ○ |
| Linux | ○ |
| Android | ○ |
| iOS | ○ |
Windows10 UWP | ○ |
WebGL (asm.js/webassembly) | ○*1 |
| Lumin (Magic Leap) | ○ |
| ChromeOS | ○ |
Support Modules
API Reference
https://enoxsoftware.github.io/OpenCVForUnity/doc/html/annotated.html
Support Modules
Windows | Mac | Linux | iOS | Android | Windows10 UWP | WebGL *2 | ChromeOS | |
|---|---|---|---|---|---|---|---|---|
| aruco | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| bgsegm | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| bioinspired | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| calib3d | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| core | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| dnn | ○ | ○ | ○ | ○ | ○ | × | ○ | ○ |
| dnn_superres | ○ | ○ | ○ | ○ | ○ | × | ○ | ○ |
| face | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| features2d | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| img_hash | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| imgcodecs | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| imgproc | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| ml | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| objdetect | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| phase_unwrapping | ○ | ○ | ○ | ○ | ○ | ○ | × | ○ |
| photo | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| plot | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| structured_light | ○ | ○ | ○ | ○ | ○ | ○ | × | ○ |
| text *3 | ○ | ○ | ○ | ○ | ○ | × | ○ | ○ |
| tracking | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| video | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| videoio | ○ | ○ | ○ | ○ | ○ | ○*1 | ○*1 | ○ |
| wechat_qrcode | ○ | ○ | ○ | ○ | ○ | × | ○ | ○ |
| xfeatures2d | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| ximgproc | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| ximgproc | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| xphoto | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
Setup Guide
Setup Manual
Setup Tutorial & Demo Video
Demo Application
Android
https://github.com/EnoxSoftware/OpenCVForUnity/releases
WebGL
https://enoxsoftware.github.io/OpenCVForUnity/webgl_example/index.html
WebGL(simd)
https://enoxsoftware.github.io/OpenCVForUnity/webgl_example_simd/index.htmlWebGL(simd, threads)
https://enoxsoftware.github.io/OpenCVForUnity/webgl_example_simd_threads/index.htmlExample Project Assets
Basic Examples
Useful examples for learning the basics of OpenCV for Unity, included in the package.
OpenCVForUnity Examples (GitHub)
Advanced Examples
Useful examples for learning OpenCV for Unity in an applied context, available in the EnoxSoftware Github.
EnoxSoftware repositories (GitHub)
CV VTuber Example
https://assetstore.unity.com/packages/templates/tutorials/cv-vtuber-example-118186
CVVTuberExample(Computer Vision Virtual YouTuber Example) is an example project of controlling 3D humanoid model (“Unity-chan!” Model) using WebCamTexture. You can control the head orientation and the facial expression of the 3D humanoid model using WebCamTexture only.
The execution of this asset is required “OpenCV for Unity” and “Dlib FaceLandmark Detector”.
FaceMask Example
https://assetstore.unity.com/packages/templates/tutorials/facemask-example-79999
This asset is an example project that maps face mask to the detected faces in an image using “OpenCV for Unity” and “Dlib FaceLandmark Detector”.
MarkerLess AR Example
https://assetstore.unity.com/packages/templates/tutorials/markerless-ar-example-77560
This asset is a Marker-Less Augmented Reality Example using the actual environment as input instead of printed square markers and display 3d model in WebCamTexture in real-time.
This project’s code is a rewrite of https://github.com/MasteringOpenCV/code/tree/master/Chapter3_MarkerlessAR using “OpenCV for Unity”.
RealTime FaceRecognition Example
https://assetstore.unity.com/packages/templates/tutorials/realtime-facerecognition-example-73233
This asset is an example project of face recognition in real time using “OpenCV for Unity”.
This project’s code is a rewrite of https://github.com/MasteringOpenCV/code/tree/master/Chapter8_FaceRecognition using “OpenCV for Unity”.
MarkerBased AR Example
https://assetstore.unity.com/packages/templates/tutorials/markerbased-ar-example-29678
This asset is an Augmented Reality Example that detect and recognize markers and display 3D model in WebCamTexture in real-time.
This project’s code is a rewrite of https://github.com/MasteringOpenCV/code/tree/master/Chapter2_iPhoneAR using “OpenCV for Unity”.
Integration Examples
Other Unity Assets
- NatDevice – Media Device API (Asset Store)
- NatCorder – Video Recording API (Asset Store)
- Vuforia (Vuforia SDK for Unity)
- GoogeVR(Cardboard) (GVR Unity SDK)
Other Gadgets
- Nreal Light (Android NRSDK Unity SDK 1.9.x)
- RealSense (Unity Wrapper for RealSense SDK 2.0)
- ZED 2 or ZED Mini stereo camera (ZED Unity plugin 3.0)
- Kinect V2 (Kinect SDK 2.0 for Unity)
- Oculus Rift DK2 (Windows)
- …successful execution of the example scene.
- Raspberry Pi 2 (Windows10 loT Core,UWP)
- …successful execution of the example scene.
- Telepathy (Android)
- …successful execution of the example scene. (Link)
- HoloLens 1 / HoloLens 2 (UWP)
Magic Leap One (Lumin)[Not supported]
FAQ
General
Support Web platform?
Since the Unity Web Player does not support the native plugin, “OpenCV for Unity” does not support “WebPlayer Platform”.
A WebGL platform was added as an alternative. (Unity 5.3 or higher)
Support Windows10 UWP?
Yes. The asset supports the UWP platform.
Asset package size is large. Is there a way to reduce?
Please remove plugin folders of non-output target platforms that are included in the package.
What is the minimum file composition required for the assets to work?
You do not necessarily have to import all the files for the asset to work. If you do not need to try the example scenes, the minimum file composition required is as follows:
Offer a free version?
Yes, we offer a free trial version of OpenCVForUnity Asset.
Is there a known problem in the WebGL platform?
We know the existence of the following problems.
- Put a file that you want to use for the Utils.getFilePathAsync() in the “Aseets/StreamingAssets/” folder. In Case of WebGL platform, you need to use the Utils.getFilePathAsync() instead of the Utils.getFilePath(). (The haarcascade_frontalface_alt.xml is for OpenCVForUnityExample’s scenes. Please copy only when necessary)
- On the WebGL platform (asm.js), the calculation result of Float type may be significantly different from other platforms. When using the OpenCV’s method that use the Mat class (CvType is CV_32F) as an argument, you need to pay attention to the calculation precision.
Setup
“DllNotFoundException: opencvforunity” is displayed on the console when run the example scene.
The plugin does not seem to be loaded correctly. Please check the setup procedure.
“ArgumentException: The output Mat object has to be of the same size” is displayed on the console when run the example scene.
After having setup Plugin, Plugin may work well when you reboot Unity.
“Level ‘Texture2DtoMatExample’ (-1) could not be loaded because it has not been added to the build settings.” is displayed on the console when run the example scene.
Please add all of “***.unity” scenes into the “Assets/OpenCVForUnity/Examples” folder to [Build Settings] – [Scene In Build].
In DetectFaceExample or WebCamTextureDetectFaceExample, red rectangle is not displayed around a face.
You might have failed to read the “haarcascade_frontalface_alt.xml”. Please confirm whether there is the “OpenCVForUnity/StreamingAssets” folder at the right position.
OpenCV
How do learn the details of OpenCV’s method and argument?
Please refer to OpenCV official document (http://docs.opencv.org/3.3.0/index.html) and OpenCV Tutorials (http://docs.opencv.org/3.3.0/d9/df8/tutorial_root.html) for the details of the argument of the method.
How can I convert Mat class operators defined in C++?
See this page. ([Tips] Way to translation of Mat class operators defined in C++)
How to a way to catch native OpenCV’s errors code?
In order to display the native opencv’s error code, please enclose the code in Utils.setDebugMode(true) and Utils.setDebugMode(false).
Utils.setDebugMode(true); ------------
Utils.setDebugMode(false);See this page for details. ([Tips] How to catch native OpenCV’s errors code (CVException handling))