Using Smooth Streaming Media Element for Windows Phone 7

In my previous post I described the avaliability of  Smooth Streaming Media Element (SSME) for Windows Phone (WP) as a part of the Integrated Media Platform vision for IIS Media Services. In this post I would show you how you can use SSME for WP7 in order to reproduce Smooth Streaming (SS) content live and ondemand. In this post I'll show you how it is possible use directly SSME in a WP project but I would remember that if you are looking for a complete player solution you can leverage the Silverlight Media Framework  that give to you a complete solution based on SSME compatible also with WP .

For first you can download the dll that contain SSME for WP from the latest version of  Smooth Streaming Client SDK . After intalled the SS Client SDK  you can found the Microsoft.Web.Media.SmoothStreaming.dll for WP7    in (your drive):\Program Files (x86)\Microsoft SDKs\IIS Smooth Streaming Client\  select the Windows Phone version.

In order to start with WP you have to download and Install Visual Studio for Windows Phone Tool and Expression Encoder for Windows Phone . You can read all the info for starting with Windows Phone  here.

If you are intrested in a global player that works in every page of your app and in a live smooth streaming audio only, you can read for more info this additional post on this topics.

You can start a new Windows Phone  project from Visual Studio using the specific template project: 

And after that you can add a reference to Microsoft.Web.Media.SmoothStreaming.dll for WP :

In order to insert the SSME in your project you have to add a reference to your MainPage.xaml page, and insert this reference in the <phone:PhoneApplicationPage tag:

 xmlns:SSME="clr-namespace:Microsoft.Web.Media.SmoothStreaming;assembly=Microsoft.Web.Media.SmoothStreaming"

I suggest to change also the Orientation Support on the page in order to have the landscape orientation that is more useful for wotching videos:

   SupportedOrientations="Landscape" Orientation="Landscape"

 this means that you have this PhoneApplicationPage tag in your mainpage.xaml:

 <phone:PhoneApplicationPage
    x:Class="WindowsPhoneSSMESample.MainPage"
    xmlns="https://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="https://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:phone="clr-namespace:Microsoft.Phone.Controls;assembly=Microsoft.Phone"
    xmlns:shell="clr-namespace:Microsoft.Phone.Shell;assembly=Microsoft.Phone"
    xmlns:d="https://schemas.microsoft.com/expression/blend/2008"
    xmlns:mc="https://schemas.openxmlformats.org/markup-compatibility/2006"
    xmlns:SSME="clr-namespace:Microsoft.Web.Media.SmoothStreaming;assembly=Microsoft.Web.Media.SmoothStreaming"
    mc:Ignorable="d" d:DesignWidth="728" d:DesignHeight="480"
  
    FontFamily="{StaticResource PhoneFontFamilyNormal}"
    FontSize="{StaticResource PhoneFontSizeNormal}"
    Foreground="{StaticResource PhoneForegroundBrush}"
    SupportedOrientations="Landscape" Orientation="Landscape"
    shell:SystemTray.IsVisible="True" Loaded="PhoneApplicationPage_Loaded">

 In this way you have referenced a namespace SSME that point to the control that are in Microsoft.Web.Media.SmoothStreaming and you could insert these in your xaml page. Now you can add in ContentPanel Grid this tag in order to insert SSME in your page:

 <!--ContentPanel - place additional content here-->
        <Grid x:Name="ContentPanel" Grid.Row="1" Margin="12,0,12,0"></Grid>
            <SSME:SmoothStreamingMediaElement x:Name="video"/>
        </Grid>

We can use  video as a reference from our code to the instance of SSME inserted in the page. From the Loaded Event in MainPage.xaml we can manage the reference and set the SmoothStreamingSource property to the uri of the manifest SS content that we want reproduce:

  video.SmoothStreamingSource = new Uri("https://localhost/adaptivestreaming/video/vc1/big_buck_bunny_1080p_surround.ism/manifest"); 

You can use Play , Stop, Pause method to control reproduction and you can subscribe all the event in the SSME like      video.CurrentStateChanged  or   video.MediaFailed or   video.ManifestReady  to manage the lifecycle of the SSME in the same way that is possible do for SSME for PC (https://msdn.microsoft.com/en-us/library/ee958035(v=VS.90).aspx) .In order to access to the SSME from the code you have to add to your .cs file this namespace:

using Microsoft.Web.Media.SmoothStreaming;

For example in order to manage Play, Pause, audio volume etc and to show status of Streaming and current Level of bitrates involved in the reproduction we can add these elements to our xaml:

   <!--LayoutRoot is the root grid where all page content is placed-->
    <Grid x:Name="LayoutRoot" Background="Transparent">
     
        <Grid.RowDefinitions>
         
            <RowDefinition Height="0.90*"></RowDefinition>
           
            <RowDefinition Height="0.10*"></RowDefinition>
           
        </Grid.RowDefinitions>
      
        <SSME:SmoothStreamingMediaElement x:Name="video" Grid.Row="0" />
       
        <StackPanel Orientation="Horizontal" Grid.Row="1">

           <Button   x:Name="PlayButton" Width="50" Click="PlayButton_Click" Loaded="PlayButton_Loaded"/>
           <Button   x:Name="StopButton" Content="Stop" Width="50" Click="StopButton_Click" />
           <TextBlock x:Name="status"/>
           <TextBlock x:Name="currentBitrate"/>
        
        </StackPanel>
      
    </Grid>

I removed the content grid and added directly in the first layaout grid in our mainpage.xaml:

- two buttons used for Play and Stop directly attached to an handler for click event in the code described before;

- two textblocks used to present status and current bitrates used in reproduction

 

On the code in .cs files I added this handler and additional code in the loaded event:

private void PhoneApplicationPage_Loaded(object sender, RoutedEventArgs e)
        {
            video.CurrentStateChanged += new RoutedEventHandler(video_CurrentStateChanged);
            video.PlaybackTrackChanged +=new EventHandler<TrackChangedEventArgs>(video_PlaybackTrackChanged);
            video.SmoothStreamingSource = new Uri("https://localhost/adaptivestreaming/CESARONI_DVD02Video.ism/manifest");  
           
        }

        void video_PlaybackTrackChanged(object sender, TrackChangedEventArgs e)
        {
            currentBitrate.Text = e.NewTrack.Bitrate.ToString();
        }

        void video_CurrentStateChanged(object sender, RoutedEventArgs e)
        {
            status.Text = video.CurrentState.ToString();
        }

        private void StopButton_Click(object sender, RoutedEventArgs e)
        {
            //This should simply stop the playback
            video.Stop();
            //We should also reflect the chang on the play button
            PlayButton.Content = "Play";
        }

        private void PlayButton_Loaded(object sender, RoutedEventArgs e)
        {
            switch (video.AutoPlay)
            {
                case false:
                    PlayButton.Content = "Play";
                    break;
                case true:
                    PlayButton.Content = "Pause";
                    break;
            }
        }

        private void PlayButton_Click(object sender, RoutedEventArgs e)
        {

            //Monitor the state of the content to determine the right action to take on this button being clicked
            //and then change the text to reflect the next action
            switch (video.CurrentState)
            {
                case SmoothStreamingMediaElementState.Playing:
                    video.Pause();
                    PlayButton.Content = "Play";
                    break;
                case SmoothStreamingMediaElementState.Stopped:
                case SmoothStreamingMediaElementState.Paused:
                    video.Play();
                    PlayButton.Content = "Pause";
                    break;
            }

        }

 As you can see before I subrcibed in Loaded event two events :

  video.CurrentStateChanged += new RoutedEventHandler(video_CurrentStateChanged);
  video.PlaybackTrackChanged +=new EventHandler<TrackChangedEventArgs>(video_PlaybackTrackChanged);

The first is used to catch change status in SSME and in the handler I write in a specifc TextBlock the current status of SSME, in the second we capture changes in bitrates level used in reproduction and also in this case in the handler I write the current level of bitrate in a specific TextBlock.

The others handlers added to the code are conected to the click event and are used to manage the play stop pause on the SSME. You could read in the code before how this works.

If you would avoid that the application is deactiveted when the Phone go idle or lock (for example you stream audio content and would that the user continue to listen the content ) , you can add to the Loaded Event this code:

PhoneApplicationService.Current.ApplicationIdleDetectionMode = IdleDetectionMode.Disabled;

that prevent deactivation of the app like you can read here and in the specific api description in sdk of wp .

With SSME for WP7 you can play also audio only streaming.  

In order to reproduce SS content with SSME on WP7 you have to pay attention to some additional rules about the type of content supported in the current version of WP7 and SSME 1.5. Here you have a list of supported codecs for SS:  VC1 / H.264 for video, WMA Pro/HEAAC/AACL for audio.

The Maximum Resolution admited on the W7 for H.264 are : 

  • 720 x 480 pixels at 30 fps
  • 720 x 576 pixels at 25 fps

The Current profile for H264 supported on the phone are baseline and main  Level 3.0. 

It is nesessary PAY ATTENTION TO THE FACT THAT IN CURRENT VERSION OF WP IN SOME HW MULTIPLE RESOLUTIONS BITRATE ISN'T SUPPORTED  this means that it is necessary have the same resolution for all the bitrates inlcuded in the range of levels to reproduce in SSME for WP7.  You can check if the device supports multiple resolutions during a play with this api https://msdn.microsoft.com/en-us/library/microsoft.phone.info.mediacapabilities.ismultiresolutionvideosupported(v=VS.92).aspx. In the case that the devices doesnt support multiple resolution and you have a previous encoded content with multiple resolutions or you want create a  single asset for multiple devices , you have to pay attention to this and in the case you can use the StreamInfo.RestrictTracks API   in a ManifestReady event , in order to restrict the range of bitrates used in reproduction . For example imagine to have a manifest that have for biterates <= of 600K the same resolutions and for bitrates >  400k a variable resolutions in order to meet quality requirements of others devices like a PC, in the case tha the  is multipleresolutionvideo supported respond with false you can add the code below in the manifest ready event on WP in order to limit the bitarates used in the reproduction :

 void video_ManifestReady(object sender, EventArgs e)
        {
            if (video == null)
            {
                return;
            }

            foreach (SegmentInfo segment in video.ManifestInfo.Segments)
            {
                foreach (StreamInfo streaminfo in segment.AvailableStreams)
                {
                    if (MediaStreamType.Video == streaminfo.Type)
                    {
                        List<TrackInfo> tracks = new List<TrackInfo>();

                        foreach (TrackInfo track in streaminfo.AvailableTracks)
                        {
                            if (track.Bitrate <= 400000)
                            {
                                tracks.Add(track);
                            }
                        }
                        streaminfo.RestrictTracks(tracks);
                    }
                }
            }
        }

You can subcribe the ManifestReady event in SSME called video in this way :  video.ManifestReady += new EventHandler<EventArgs>(video_ManifestReady);

In alternative you can create an adhoc manifest, server side , one for every devices and point every devices to the right manifest.

Propose an encoding profile is difficult because there are a lot of parameters to consider like bandwith avaliable type of content etc, but on my experience in others project that leverare SS for WP7 I can suggest to have in the manifest of your content this generic profile:

  Use Main Level 3.0  with 640x352 for 16:9 with this levels: 1000 k(Full framerate but <=30) , 800k (Full framerate but <=30) , 600k (Full framerate but <=30) , 400k (Full framerate but <=30) , 200k (half framerate <=15), 100 k(half framerate <=15)

If you would serve the same content to WP7, PC and Apple Devices like iPhone and iPad leveraging the capability of IIS Media Services 4.0 and IIS Transform Manager , you have to built a profile using H264 Baseline Level 3.0 and pay attention to the resolutions in differents bitrates levels and Transmux the SS also in Apple Adaptive Streaming.

If you have to stream a to low level of the maximu 1000k suggested before , you can leverage these profiles for WP7 and iPhone/iPad and in the case add more level fo PC using variable resolution for profiles > 600k and use RestrictTracks API on PC for the minimu bitrates for WP7 for the maximum bitrate and during the transmux operation on IIS Transform Manager and IIS MS 4,0 (in the case of live streaming ) limiting profile fo Apple devices to 600k :

4:3 use Baseline Level 3.0  with 480x360 with this levels: 600 k(Full framerate but <=30) ,  400k (Full framerate but <=30) , 200k (half framerate <=15), 100 k(half framerate <=15)

 16:9 use Baseline Level 3.0  with 480x272 with this levels: 600 k(Full framerate but <=30) ,  400k (Full framerate but <=30) , 200k (half framerate <=15), 100 k(half framerate <=15)

 If you don't need to cover old iPhone you could move to Main Level 3.0 with the same resolution in order to increase the quality.

Pay attention to the fact that  profiles suggested before are only a sample and not necessary these profile are good for your scenarios. It is necessary test it  and adjust it to your requirements in order to have the best solution.