OpenCV: building UWP binaries
[This blog article documents my continuing mission to figure out how to use OpenCV in UWP apps...]
How to build OpenCV native dlls for UWP
OpenCV is on GitHub. There are three relevant forks/branches, as of writing on November 2015:
[1] https://github.com/itseez/opencv
+- [2] fork https://github.com/Microsoft/opencv
+- [3] branch https://github.com/Microsoft/opencv/tree/vs2015-samples
Repository [1] is the master. In it, the native OpenCV binaries are built for all different platforms using CMake. As far as I can tell, this repository only has support via CMake for building Windows Store and Windows Phone 8.0 and 8.1 apps. It doesn't seem to have UWP. And it also requires you to install some particular old versions of VS, as per its readme.txt.
Fork [2] doesn't seem to bring anything new.
Branch [3] contains Visual Studio .sln files from which you can build UWP versions of all the native OpenCV binaries: x86, x64 and ARM, Debug and Release. The Release builds actually were a bit wonky, but I submitted PR #69 to make them build right. So: if you clone this repository, select this branch, set an environment variable, and open up vs2015/WS/10.0/ARM/OpenCV.sln, or x86/OpenCV.sln, or x64/OpenCV.sln, then you can successfully build UWP binaries for OpenCV. This will produce 15 binaries for each architecture+build combination.
How to use OpenCV native dlls for UWP
I tried out the OpenCV native dlls in a C++ UWP app, and they worked fine.
More interestingly, the dlls all together add up to about 30mb, but I was able to use just three of them that I needed and they worked fine. This satisfies one of my central requirements, to be able to deploy just a cut-down version of OpenCV with my store app, to keep size down.
I did File>New>C++>Windows>UWP blank app. In Solution Explorer I added three DLLs as "content" files in the root directory of my project so they would be copied into the resulting appx, adjacent to the .exe:
opencv_core300d.dll
opencv_igmcodecs300d.dll
opencv_igmproc300d.dll
In Solution Explorer I added three import-libraries. By default, when you add LIBs to a C++ project, they are taken as import libraries:
opencv_core300d.lib
opencv_igmcodecs300d.lib
opencv_igmproc300d.lib
In Project Properties I added an include directory to my C++ project options
vs2015/WS/10.0/include
In my mainpage C++ file, I added a bunch of #include and using statements:
#include <robuffer.h>
#include <opencv2\imgcodecs.hpp>
#include <opencv2\imgproc.hpp>
using namespace Windows::UI::Xaml::Media::Imaging;
using namespace Windows::Storage::Streams;
using namespace Microsoft::WRL;
I added a content file "demo.jpg" in Solution Explorer, and in MainPage.xaml I added an Image control named "image1", and added these lines in OnNavigatedTo in my mainpage C++ file:
cv::Mat bmp1 = cv::imread("demo.jpg");
cv::Mat bmp2 = cv::Mat(bmp1.rows, bmp1.cols, CV_8UC4);
cv::cvtColor(bmp1, bmp2, CV_BGR2BGRA);
WriteableBitmap^ wbmp = ref new WriteableBitmap(bmp2.cols, bmp2.rows);
IBuffer^ buffer = wbmp->PixelBuffer;
unsigned char* dstPixels;
ComPtr<IBufferByteAccess> pBufferByteAccess;
ComPtr<IInspectable> pBuffer((IInspectable*)buffer);
pBuffer.As(&pBufferByteAccess);
pBufferByteAccess->Buffer(&dstPixels);
memcpy(dstPixels, bmp2.data, bmp2.step.buf[1] * bmp2.cols * bmp2.rows);
image1->Source = wbmp;
Result: it all worked. The image displayed correctly.
Next Steps
I want to package each individual DLL up as a NuGet package, for 15 NuGet packages in total. Each package should include x86+x64+ARM versions of the DLLs, and corresponding import libraries. It should also include the include files pertaining to that DLL. That way, as a C++ author, I will be able to bring in only the particular DLLs that I want. To do this I'll have to figure out the dependencies between the DLLs, and the dependencies between header files, and then express these as NuGet dependencies. (I don't yet know how to create NuGet packages for C++. I'll look to "Win2d.uwp" as the canonical example. I don't know if each package should also include debug builds, but I suspect not. I don't know where PDBs should go.)
I think that NuGet generation should be done as part of [3] branch vs2015-samples. This is the branch that has .sln files for building all the DLLs, so it's appropriate for it to also include NuGet-package-building. I don't know if this is within the vision of the owners of that branch or not. Ultimately, in the future, I don't know if the owners of Itseez/opencv would be open to their CMake system building the NuGet packages as well?
Beyond that, the major issue is that OpenCV has a C++ API. This is not friendly to pinvoke interop with .NET. OpenCV used to have a plain C API which is friendly, but the C API has been deprecated and is not getting new features. I don't wish to invent a pinvoke-friendly wrapper around the OpenCV C++ API.
Approach (1): I'd like to test the scenario of a hybrid app, where most of the app is in .NET, and it calls into a C++ winrt component that I author, and this C++ component does all the calls into OpenCV, which it obtains via NuGet.
Approach (2): OpenCVSharp has a BSD license and already contains a set of 'extern "C"' wrappers around the C++ API. The ideal would be for OpenCVSharp to be able to consume my native UWP dlls for OpenCV. There already is a "dll-free" version of OpenCVSharp, which merely contains the pinvoke wrappers, and expects you to supply the OpenCV DLLs some other way. There are a few unknowns with this approach. I don't know if the OpenCVSharp project has different version of OpenCV (and hence different include files) as compared to the one in branch [3]. I don't know if the native extern "C" assembly that comes with OpenCVSharp will be compatible with the DLLs I build, and the fact that there are 15 of them rather than just 1 big "world" dll. I don't know if it will need to switch to "delay-load" to support the scenario where you only package say 3 of the OpenCV dlls with your store app.
Notes
I am hugely excited! It was thrilling to see the UWP versions of all the DLLs get built properly. I think I know how to package up DLLs into NuGet better than most folk, and I can do a best-in-class job. I'm excited that I was able to get something working end-to-end at least in C++.