Kinect and UWP: new options in the Windows 10 Anniversary Update
UWP apps today cannot access most of the data from a Kinect sensor. This post provides an overview of what we plan to deliver in the near future to begin bringing Kinect for Windows into the Universal Windows Platform (UWP) world.
With the Windows 10 Anniversary Update and a Kinect driver update, Kinect developers will be able to read RGB, infrared (IR), and depth data from the Kinect sensor. The Kinect sensor’s microphone array is already visible to the UWP audio and speech recognition APIs today (although not all features from the Kinect for Windows SDK are accessible to UWP APIs)—this remains unchanged.
Skeleton data is important to many Kinect apps. Through a supplemental SDK that will be available later this year, UWP developers will have access to skeletal poses. With the new capabilities in the Windows 10 Anniversary Update, developers can also use the existing Windows.Media.FaceAnalysis APIs with Kinect to detect and track faces.
Existing Kinect Win32 apps and SDK capabilities will not be affected by this new UWP support.
Details: how it works
Today, there are several sensors that can provide rich correlated data, such as RGB, IR, or depth information. To make such data available to app developers in a device-independent manner, we introduced the Windows.Media.Capture.Frames APIs, a set of extensions to Media Capture that adds frame-by-frame access to RGB, IR, and depth data, as well as sensor correlation to the traditional image and video capture features, all with a consistent, familiar programming model. In addition, this extended model can give UWP apps access to custom streams from a sensor.
In the case of Kinect for Windows, we plan to make use of the custom streams capability to deliver skeleton data from Kinect to UWP apps. This will require a small supplemental Kinect-specific SDK to decode the custom streams on the app side, which we’ll make available later this year.
In order to make use of the new frame-by-frame capabilities of Media Capture, developers need a device capable of delivering rich data streams, as well as an updated driver. A driver update for Kinect will be available later this spring.
By using the proven Media Foundation driver architecture, independent hardware vendors (IHVs) will be able to readily connect their own devices—including RGB, IR, depth, and other custom sensors—to the same Media Foundation infrastructure that now powers Kinect.
The new API, together with the extensible driver model, will empower developers to target Kinect and supported third-party sensors in a hardware-agnostic way to gain better understanding of the world around them and the people in it.
In summary, these are the elements that enable UWP apps to access Kinect:
- The new Windows.Media.Capture.Frames extensions to Media Capture are available in the Windows 10 SDK preview, build 14332 or newer. These API extensions are device independent. To access IR and depth data, a compatible device and a matching device driver are also required.
- A matching driver update for Kinect will be available later this spring.
- To access skeleton data from Kinect, which is delivered through Windows.Media.Capture as a custom stream, a Kinect-specific supplemental SDK is needed, which we plan to release in the second half of 2016.
Join the Windows Insider Program so that you can access Windows 10 build 14332 and its preview SDK. Watch the Channel 9 video for a brief preview of the full capabilities that will be available when the Windows 10 Anniversary SDK and the supplemental Kinect SDK are released to the public, including full skeleton support. The video also tours the code for a simple demo app that demonstrates how to get frames from Kinect or any other RGB camera on your system.
The Kinect for Windows Team
Key links