Clap and the filter graph claps with you


One of my colleagues was a fount of ideas, some of them crazy, some of them clever, and some of them both. I think this one counts as both.

To render multimedia content with DirectShow, you build a so-called filter graph. A filter graph represents a series of transformations that are applied to data as it travels through the graph. For example, bytes from a video file may go through a splitter filter which separates the audio from the video data, then the two data streams each go through a respective audio and video decoder, which converts the compressed data into uncompressed sound or video data, and then to a render device which actually plays the sound samples out the speaker or puts the video image on the screen.

One of the components of a filter graph is the reference clock. The purpose of the reference clock is to keep track of time, so that all the parts of the filter graph produce their output in a synchronized manner. If there weren't something keeping track of this, the audio would fall out of sync with the video.

My colleague decided to create an unusual reference clock. Whereas most reference clocks are based off of real-world time, my colleague's reference clock started by taking input from the microphone. It did some analysis of the volume of the incoming sound stream and detected when there were loud impulses of sound. It then generated a stream of clock ticks based on the rate at which those impulses were detected.

In other words, the clock was driven by clapping.

I was lucky enough to be one of the first people to see this crazy reference timer in action. First, you started up the application and picked a video clip. The video clip just sat there. As you started clapping, the video clip started playing. If you clapped at about 80 beats per minute, the video clip played at its normal speed. If you clapped faster, the video clip ran faster. If you clapped slower, the video clip ran slower. If you stopped clapping, the video clip stopped.

It was freaky cool.

Totally useless, but freaky cool.

Update: Commenter hexatron pointed out that the Metronome sample does the same thing as my colleague's crazy reference clock. I don't know if it's literally the same code, but it's functionally equivalent.

Comments (19)
  1. pcooper says:

    I don’t suppose that the code could be released that does this?

  2. Ami Cusbreaf says:

    In other news, Raymond’s friend received a subpeona from patent attorneys representing the makers of The Clapper.  They claim their patent on turning devices on and off through a clap of the hands also covers controlling audio and video playback through software.  They are demanding that he cease and desist having crazy and clever, ummm… new and innovative, ideas, reimburse them for all legal fees, and fork over his life savings.

  3. Nathan_works says:

    I wonder if this could be setup with a feedback loop for bad sit-coms, where the laugh-track is what drives the playing of the show (versus clapping)..

    Not that it would make the shows any more entertaining, but the feedback loop would be interesting..

    Or tie it into watching something like "The Price is Right" where there is much audience clapping..

  4. John says:

    I think it would be more fun to yell profanities into the microphone, but maybe that’s just me.

  5. ton says:

    Brilliant but a sad waste of talent…

  6. RobO says:

    How about playing it the drum solo from In-A-Gadda-Da-Vida?

    Maybe you could use it to force one music video to play to the beat of another song, like playing Everybody Wants to Rule the World with Ailen Ant Farm’s Smooth Criminal as the reference timer.

  7. someone else says:

    I expect this in the next (DirectX|Platform) SDK.

  8. Paul Williams says:

    That is way cool.  Thank you for sharing.

  9. keith says:

    Is there a reference work that this was designed to — like mp3 compression and Suzanne Vega’s "Tom’s Diner", or the 74 minute length of a CD and a specific performance of Beethoven’s 9th?

  10. hexatron says:

    The Metronome example of the Direct X SDK makes writing a new reference clock extremely simple. I used it to sync a movie to some other onscreen animation.

    Connecting the filter properly is harder than writing it (at least in C++, where each pin seems to be about 20 lines of tedious code).

  11. Jolyon Smith says:

    In the music gadgets biz (keyboards, recording, FX etc), this sort of thing is called "Tap Tempo", although admittedly it usually involves tapping a button or pad (hence: "TAP Tempo"), rather than clapping, but the principle is the same.

    Even with video I can see that the ability to set/adjust tempo rythmically could be very useful if trying to achieve a beat synergy (as opposed to precise "synchronization") between some piece of video and soundtrack audio (during the mixing stage).

  12. Heads up for your friend and his crazy thoughts, That’s one hell of an idea he got there lol.

  13. Worf says:

    Hrm…

    If that reference clock was a DirectShow object, you could possibly use it with the DirectShow Graph Builder program…

    I never played much with DirectShow other than fight with it trying to get my camera source to work. That might make for interesting "stop motion" effects with a wonky reference clock. Though, camera sources are often special in that generate the reference clock so you can tell if you’re dropping frames or falling behind.

  14. Gabest says:

    As hexatron already mentioned it, it is the metronom sample. For those who were looking for the source.

  15. Miral says:

    It’d be cool if this could be downloaded from somewhere.  Can you ask your colleague if they’re amenable?

    (Sure, it’s not very *practical*.  But it’d be fun to play with.)

  16. PinkDuck says:

    Why did Raymond choose to use "based off of" instead of "based on" or "based upon"?

  17. someone else says:

    Holy crap, this is great!

    For those trying it out, my psychic powers tell me that the nonexistent README.TXT (what is this, open source??) instructs you set the microphone as the audio recording device.

  18. James Schend says:

    PinkDuck:

    He’s in Washington. "Based off of" is what’s said here, probably a regional thing. "Based on" would be acceptable, but "based upon" would just sound weird, or snobby.

  19. Caliban Darklock says:

    I totally want this. It would be a great way to synchronise video to percussion. I wonder if something similar was used in that dancer thingy?

Comments are closed.