Annotating Video with Ink using Silverlight

Silverlight makes delivering video content in the browser very easy. Silverlight also has ink support built-in. So why not combine these two features to enable some great new scenarios?

I have put together a little sample using Silverlight v1.0 to record and play back ink annotations, synchronized with the video playback. Below is the key piece of XAML markup for this application. We are using an InkPresenter that hosts the MediaElement with the video and takes care of the ink renderering. Also note the empty storyboard here as this is the magic sauce for the synchronized ink playback.

<InkPresenter Name='inkPresenter' Canvas.Left='10' Canvas.Top='10'

MouseLeftButtonDown='onInkPresenterDown'

MouseMove='onInkPresenterMove'

MouseLeftButtonUp='onInkPresenterUp'>

<InkPresenter.Resources>

<Storyboard Duration="0:0:0" Completed="onStrokePlaybackTimerTick"

x:Name="strokePlaybackTimer" />

</InkPresenter.Resources>

<MediaElement Name='mediaElement' Source='bear.wmv'

Width='720' Height='480'

AutoPlay='False' MediaEnded='onMediaEnded'/>

</InkPresenter>

The relevant pieces in the JavaScript code behind the XAML (in file events.js), are the following functions:

  • ConvertInkToString(strokes) - a function to serialize the point data for the ink strokes into a string for persistence purposes
  • ConvertStringToInk(serializedStylusPoints) - the reverse function that returns a StrokeCollection for a serialized ink string
  • onStrokePlaybackTimerTick() - during playback this function is called on every frame; when it is time to play back a stroke, we do so point by point

The full source code is attached to this post. I have also uploaded the sample to the Silverlight Streaming service, so I can pull it directly into my blog here. To check it out, just click the button below to load the sample. Once the video is loaded, you can click the 'Playback' button to replay my pre-recorded ink annotation - or you can click 'Start Recording' to record your own annotations, and then play them back afterwards.

VideoAnnotation.zip