Remix the music with Kinect


This spring, visitors to Seattle radio station KEXP’s new home are being treated to the ultimate air-guitar experience: as they pass through a 4,000-square-foot art installation, their movements alter the music of some of the station’s most popular artists.

This innovative project—dubbed a “virtual remix”—uses Kinect for Windows to capture the visitors’ body movements. The audio output, as well as the visual accompaniments to the music, both respond to visitors’ movements in an interactive and ever-evolving environment.

As people move through the installation, they continuously reshape and customize the music of alt-J, Floating Points, Ólafur Arnalds, and Samaris, blurring the line between creator and audience.

The installation is powered by four Kinect for Xbox One sensors mounted behind highly flexible mesh curtains hung in the corners of the installation space. Each sensor links to its own PC running Windows 8.1. The mesh curtains appear opaque to the Kinect sensors’ depth sensing technology but invisible to the sensors’ color camera and infrared emitter. The fabric is also pleasant to the touch, which encourages visitors to push on it. As they do so, the Kinect depth cameras collect data on how far and in what direction (left, right, up, or down) the mesh is deformed.

These data are then translated into alterations to the music in real time. The music tracks had been previously split into various stems: the drum stem, the bassline stem, the melody stem, and the vocal stem. The Kinect depth data alters one or more stems, depending on how hard and in what direction a visitor pushes on the mesh fabric. These acoustical alterations are complex. A visitor’s push might change both the volume and the components of the drum stem; for example, making the high hat drum component more pronounced. The depth data also trigger changes in the ambient lighting, which is controlled by TouchDesigner.

With multiple visitors pushing on different parts of the mesh curtains, the experience becomes even more complex, as the actions of each visitor alters the musical components and the accompanying lighting. And since the fabric is highly elastic, a visitor can push his or her entire body into a curtain, creating a large deformity in its shape. In fact, groups of visitors can push together into the fabric, generating huge deformities. The developers used the size of deformities to drive a lot of the musical interactions, reasoning that large deformities indicate a high level of enthusiasm or playfulness on the part of visitors.

Given the large size of the installation, the developers had to set it up and test it in pieces, in a limited amount of time. Kinect Studio made this feasible, as it enabled the developers to record each session and then debug the coding later, outside of the live test environment. The developers also made heavy use of Kinect Common Bridge, an open-source application that let them work more quickly with Kinect for Windows.

“Inside the Music of KEXP: an Interactive Music Experience” is part of Microsoft’s Music x Technology program, which explores emerging modes of artistic expression and celebrates advances in interactive technology. The KEXP installation premiered on April 16, the grand opening of the station’s new home, and runs through May 8. So if you’re in Seattle in the next few days and want to get out of the rain, stop on by and join the remix.

The Kinect for Windows Team

Key links

Comments (3)

  1. Edgar Sarmiento says:

    What happens with the kinect for windows V2 device that even no support for universal applications ( UWP ) on windows operating system 10. However, since users pressed to make the operating system upgrade . I am developer and try to run an app on Windows 10 UWP kinect does not recognize the gestures and movements of the user.

    1. Hi,
      Please check out our recent blog post for an overview of what we plan to deliver in the near future to begin bringing Kinect for Windows into the Universal Windows Platform (UWP) world:

      https://blogs.msdn.microsoft.com/kinectforwindows/2016/05/13/kinect-and-uwp-new-options-in-the-windows-10-anniversary-update/

      Also, be sure to check out the public Kinect for Windows v2 SDK forum, where you can exchange ideas with the Kinect community and Microsoft engineers. You can browse existing topics or ask a new question by clicking the “Ask a question” button on the forum webpage.

      Access the forum at http://aka.ms/k4wv2forum.

  2. Attendees at Moogfest, an annual celebration of the synthesis of music, art, and technology, will be the next to enjoy this Kinect-enabled experience, as they manipulate a Grimes song with their body movements.

Skip to main content