This spring, visitors to Seattle radio station KEXP’s new home are being treated to the ultimate air-guitar experience: as they pass through a 4,000-square-foot art installation, their movements alter the music of some of the station’s most popular artists.
This innovative project—dubbed a “virtual remix”—uses Kinect for Windows to capture the visitors’ body movements. The audio output, as well as the visual accompaniments to the music, both respond to visitors’ movements in an interactive and ever-evolving environment.
As people move through the installation, they continuously reshape and customize the music of alt-J, Floating Points, Ólafur Arnalds, and Samaris, blurring the line between creator and audience.
The installation is powered by four Kinect for Xbox One sensors mounted behind highly flexible mesh curtains hung in the corners of the installation space. Each sensor links to its own PC running Windows 8.1. The mesh curtains appear opaque to the Kinect sensors’ depth sensing technology but invisible to the sensors’ color camera and infrared emitter. The fabric is also pleasant to the touch, which encourages visitors to push on it. As they do so, the Kinect depth cameras collect data on how far and in what direction (left, right, up, or down) the mesh is deformed.
These data are then translated into alterations to the music in real time. The music tracks had been previously split into various stems: the drum stem, the bassline stem, the melody stem, and the vocal stem. The Kinect depth data alters one or more stems, depending on how hard and in what direction a visitor pushes on the mesh fabric. These acoustical alterations are complex. A visitor’s push might change both the volume and the components of the drum stem; for example, making the high hat drum component more pronounced. The depth data also trigger changes in the ambient lighting, which is controlled by TouchDesigner.
With multiple visitors pushing on different parts of the mesh curtains, the experience becomes even more complex, as the actions of each visitor alters the musical components and the accompanying lighting. And since the fabric is highly elastic, a visitor can push his or her entire body into a curtain, creating a large deformity in its shape. In fact, groups of visitors can push together into the fabric, generating huge deformities. The developers used the size of deformities to drive a lot of the musical interactions, reasoning that large deformities indicate a high level of enthusiasm or playfulness on the part of visitors.
Given the large size of the installation, the developers had to set it up and test it in pieces, in a limited amount of time. Kinect Studio made this feasible, as it enabled the developers to record each session and then debug the coding later, outside of the live test environment. The developers also made heavy use of Kinect Common Bridge, an open-source application that let them work more quickly with Kinect for Windows.
“Inside the Music of KEXP: an Interactive Music Experience” is part of Microsoft’s Music x Technology program, which explores emerging modes of artistic expression and celebrates advances in interactive technology. The KEXP installation premiered on April 16, the grand opening of the station’s new home, and runs through May 8. So if you’re in Seattle in the next few days and want to get out of the rain, stop on by and join the remix.
The Kinect for Windows Team