Kinect and the self-driving car

You’re driving along in the right lane of a busy four-lane street, minding your speed and obeying all the rules of the road. Up ahead you spot an obstacle—two sandbags surrounding a storm drain. You slow down and check the traffic in the lane to your left, where you see a city bus lumbering along, some distance behind you but gradually getting closer. Now you have a choice: swing into the left lane in front of the oncoming bus or continue in your present lane and run over a sandbag.

Being an experienced driver, you choose the second option. You know that the bus will have a hard time slowing down, so cutting in front of it is inviting disaster. You also know that bumping over a sandbag or two won’t damage your car.

Unfortunately, when a self-driving car encountered such a situation this past February, it chose the first option. The car pulled into the bus' lane, with the predictable fender-bender outcome.

Unfortunate result of self-driving car's encounter with a bus

Unfortunate result of self-driving car's encounter with a bus

Could a Kinect sensor have helped the autonomous car make the better decision? Researchers at the University of La Laguna in the Canary Islands think so. They are building a system that uses the Kinect sensor’s infrared depth camera to help self-driving cars identify nearby obstacles and maneuver around or over them.

Using a self-driving golf cart, the researchers tested the Kinect depth camera against a laser rangefinder and stereo cameras. By operating the vehicle on a test road that contained stairs, ramps, and curbs, they discovered that the Kinect sensor outperformed the other two devices in detecting and correctly discerning nearby, close-to-the-ground objects. For instance, when confronted by a ramp, the laser rangefinder mistakenly determined that it was too steep to drive up. The laser device also failed to spot lower stairs, while the stereo camera had trouble identifying very close (and very far) objects and gave frequent false detections.

The Kinect sensor correctly identified the ramp as navigable and consistently outperformed the stereo cameras in detecting obstacles close to the ground. Lead researcher Javier Hernandez-Aceituno praised the Kinect sensor’s superior abilities in detecting close-range obstacles, telling reporters that the Kinect sensor allows an autonomous vehicle to navigate safely in areas where the other systems fail. And isn’t that reassuring to both future passengers and pedestrians?

The Kinect for Windows Team

Key links

Comments (5)
  1. Kellen says:

    Very cool. I’ll look forward to reading his paper to see how he filtered out all the solar IR interference. It sounds like he made a lot of progress.

    I tried crude attempt at this on a hack day at HERE, but didn’t get a very large range due to IR interference. Here’s a video with all 3d points rendered with a rough approximation of color filled in from the rgb sensor:

  2. Filip Dextré says:

    Don’t get me wrong. I love the kinect and I’m a microsoft fan; but this is an *abused* marketing story. A kinect loses a lot of info when used outdoors (low powered infrared emitter), has a way too high latency and is not industrial built. That the basic technology of a fixed (laser based) IR camera makes sense (other then LIDAR with moving lasers). See also the Tesla approach. But hinting that the kinetic as a product now can be used in self driving cars ? Please try to keep the standards a bit high here ….

  3. Jeff says:

    what if the low obstacle is a child laying on the ground, would it drove over it?

  4. Roland says:

    But will Microsoft release an industrial, sunlight immune version of the Kinect sensor, for use in such scenarios as this?

    1. Thanks for contacting us. The purpose of this blog post was to share an example of some of the interesting ways that people are using Kinect for Windows technology. The Kinect sensor wasn’t developed for this specific purpose, but we hope that the story helps prompt other ideas of how you might use Kinect for Windows technology in innovative ways.

      If you have technical questions about developing with Kinect for Windows, visit our public Kinect for Windows v2 SDK forum, where you can exchange ideas with the Kinect community and Microsoft engineers. In the forum, you can also browse existing questions or ask a new question by clicking the “Ask a question” button on the forum webpage.

      Access the forum at

Comments are closed.

Skip to main content