Scrubbing with Thumbnails with the SMF Player

Article
01/22/2010

One of the developers from one of my media customers asked me how they would implement the thumbnail preview while scrubbing in a Silverlight video player. He was turned onto the idea from seeing that feature in the Netflix Watch Instantly Silverlight player.

I thought that would be something that others could use, so I decided to use Silverlight Media Framework player to demonstrate it. It turns out that Expression Encoder has the capability to create thumbnail images from a video during encoding. You can use this capability to just create images of a video at regular intervals.

Creating the Thumbnail Images

In Expression Encoder, open the source video file and set the output format to Apply Source Encoding Settings. This will allow you to create just thumbnails.
In the Metadata tab, add one marker at one second and check the thumbnail checkbox.

Save the job as an .xej file and open it up in a text editor – I then copied the <Marker> element 29 times, incrementing the seconds for each one. Now, it would be a trivial developer task to generate the XML here for any arbitrary length video and then use the .xej job file as a template for generating jobs for any length of file with a regular interval. I did 1-second interval, but I could think that TV shows or movies would do 5-15 second intervals (depending on the genre and length).

 <?xml version="1.0" encoding="utf-16"?>
<!--Created with Expression Encoder version 3.0.1332.0-->
<JobFile
  Version="3.0">
  <Job
    OutputDirectory="C:\Users\mischero\Documents\Expression\Expression Encoder\Output"
    JobId="MISCHERO1 1-21-2010 2.28.38 PM">
    <MediaFiles>
      <MediaFile
        Source="C:\Users\Public\Videos\Sample Videos\Wildlife.wmv"
        MarkerThumbnailJpegCompression="100">
        <OutputFormat>
          <WindowsMediaOutputFormat
            AudioProfile="SourceProfile"
            VideoProfile="SourceProfile" />
        </OutputFormat>
        <Markers>
          <Marker
            Time="00:00:01"
            Value=""
            GenerateKeyFrame="True"
            GenerateThumbnail="True" />
        </Markers>
      </MediaFile>
    </MediaFiles>
  </Job>
</JobFile>

Once I had this file edited, Expression encoder created the thumbnails in less than a minute with predictable names coded to the times of the thumbnails:
The next step was to add the thumbnails to a player in a general, reproducible way. I started by creating a new project in Expression Blend and adding references for the Smooth Streaming Media Element and the Silverlight Media Framework Player. I wanted to use this as a starting point because it already had a fully skinnable design and a scrubbing slider bar.
I added the Player class to the new Silverlight application and then started editing the control Template (right-click, Edit a Template…, Edit a copy…)
In the template, I added a Border, a Grid, and three images to show the thumbnail images when scrubbing. I also gave the Border and the three Images names so I could easily reference them in the code behind.
Initially I wanted to implement this by creating a behavior but since the events (mouse moving while scrubbing) that I wanted to handle were locked up inside the control template, I had to derive a class from Player, ThumbnailPlayer.
I also saw that the Player Control template had a Scrubber control which did have the events that I wanted to handle, ScrubStarted, ScrubCompleted, and MouseMove.
In the ThumbnailPlayer control, I added two properties, ThumbnailLocation, and ThumbnailInterval which would be used to identify where the thumbnails could be found and how many seconds between image.
Once the Media file opens, I could get the duration of it and then use a WebClient to download the thumbnail images. I wanted to cache the images locally in a Dictionary<int, BitmapImage> so it would be very responsive.
When the user started scrubbing, I made the thumbnails visible and hid them when the user stopped scrubbing.

In the scrub head MouseMove handler, I then found the nearest images to the current play head and set the thumbnail images to those images:

         private void positionElement_MouseMove(object sender, MouseEventArgs e)
        {
            if (this.positionElement.IsDragging)
            {
                if (this.ThumbnailInterval <= 0)
                {
                    return;
                }

                var seconds = System.Convert.ToInt32(Math.Floor(positionElement.Value + 0.5));

                var image2 = seconds - seconds % this.ThumbnailInterval;
                var image1 = image2 - this.ThumbnailInterval;
                var image3 = image2 + this.ThumbnailInterval;

                this.SetThumbnailImage(this.thumbnail1, image1);

                this.SetThumbnailImage(this.thumbnail2, image2);

                this.SetThumbnailImage(this.thumbnail3, image3);
            }
        }

        private void SetThumbnailImage(Image image, int index)
        {
            if (image == null)
            {
                return;
            }

            BitmapImage bitmap;

            if (this.thumbnailImages.TryGetValue(index, out bitmap))
            {
                image.Source = bitmap;
            }
        }

The last thing I had to do was replace the Player class with my ThumbnailPlayer class in the page XAML:

 <local:ThumbnailPlayer x:Name="Player" Content="Player" Margin="8" 
    Style="{StaticResource ThumbnailPlayerStyle}" 
    ThumbnailLocation="Thumbnails/Wildlife" ThumbnailInterval="1">
    <local:ThumbnailPlayer.MediaElement>
        <Microsoft_SilverlightMediaFramework_Player:CoreSmoothStreamingMediaElement 
           Source=https://localhost:6801/ThumbnailDemoSite/ClientBin/Wildlife.wmv 
           AutoPlay="False"/>
    </local:ThumbnailPlayer.MediaElement>
</local:ThumbnailPlayer>

The end result is a player with thumbnails. You can download the source code here.

Updated 1/22/2010 - You can now see a demo here.

Scrubbing with Thumbnails with the SMF Player

Creating the Thumbnail Images

Additional resources