Video Telemetry – A Solution in Azure


ABSTRACT

We present a Video Telemetry solution in Azure, based on Application Insights. This solution can reach all major client platforms (iOS, Android, Windows 10 and all major web browsers, including but not limited to Azure Media Player) via various SDKs. Video Telemetry data can be accessed via Azure portal, Power BI and can also be exported to Azure storage via Continuous Export.

Introduction

Microsoft offers an expansive platform for building online video streaming/multi-screen solutions, consisting of the following:

Video Technology Category Microsoft Offering
Video Processing/Streaming Services Azure Media Services
CDN Azure CDN
DRM PlayReady Product Suite
Native Client Windows 10 media stack including

Browser Client
Console Client Xbox

 

However, we do not yet have an offering for Video Telemetry/user analytics which can reach all of the popular client platforms. Today, our customers are using 3rd party products at different price ranges. For example:

  1. High-end video telemetry, such as Conviva, as used in NBC Sports/Olympics solution;
  2. Mid-range video telemetry, such as Google Analytics Premium;
  3. Low-end/free video telemetry, such as Google Analytics.

We present such a solution powered by Microsoft Azure platform. This document provides a summary of the requirements, design and how-to.

Pricing

This solution consists of client SDKs, Azure backend and Video Telemetry code file. The Client SDKs are Application Insights Client SDKs targeting various platforms, which are open sourced. For its backend in Azure, like everything offered in Azure, it does not require licensing, totally Azure consumption based. Customers pay on Application Insights consumption. Application Insights price info can be found here.

If you use Continuous Export feature to export Video Telemetry data for long term retention and/or analysis, data storage cost would be separate.

How to Use this Doc?

If all you want is to add Video Telemetry to your Azure Media Player app, you can skip directly to the section “How Do I Use it?

 

Requirements and Approach

A video telemetry/user analytics solution needs to meet the following high level requirements:

Requirement Details
Able to COLLECT ANY data a client can CAPTURE 2 things:

  1. CAPTURE by client device: client device needs to have the capability to capture the desired data;
  2. COLLECT by such video telemetry solution: client app needs to collect the desired data as long as they can be captured.
Reach most major client platforms/languagesDev community
  1. iOS
  2. Android
  3. Browsers (supports Azure Media Player, but NOT limited to it)
  4. Windows 10
  5. Server analytics (e.g. multi-DRM license server analytics): ASP.NET, Java, PHP, Python
Portal Easy to use portal for data visualization
Tool for advanced purpose Tool for ad hoc query for deeper BI analysis
Low latency Either low latency or near-zero latency, depending on positioning:

  1. High-end video telemetry, such as Conviva;
  2. Mid-end video telemetry, such as Google Analytics Premium;
  3. Low-end video telemetry, such as Google Analytics.
Support, road map and developer community
  1. Enterprise grade support
  2. Long term road map
  3. Developer community

 

Application Insights is an extensible analytics service that monitors a live application. It processes 2.5 billion events and messages daily, from half a million applications, including some Microsoft services like Outlook and OneDrive. Application Insights is currently in Public Preview.

Application Insights documentation can be found here. In this document, we do not assume readers are experienced with Application Insights.

Application Insights technology suites provides the following with respect to the above requirements:

Requirement Currently Supported by App Insights? AI Support Details
Able to COLLECT any data a client can CAPTURE Yes AI can define and COLLECT any custom event, custom metric, with optional custom attributes, as long as client device can CAPTURE the data for collection. There is only limit on what can be captured and no limit on what can be collected by AI.
Reach most major client platforms/languages Yes
Portals for data visualization Yes
Ad hoc data exploration tool Yes Please contact the Microsoft account team for info on this tool.
Low latency Yes Approximately 3 min latency
Support, road map and developer community Yes
  • Supported by Azure Support;
  • Backed by Visual Studio Team;
  • Community contributions
  • The source code for SDKs on all platforms is Open sourced;

 

A video telemetry solution must have the following components:

  1. SDKs for all major client platforms such as iOS, Android, Windows and browsers;
  2. Infrastructure: Data collection, processing, storage;
  3. UI: Data reporting such as portal and BI tool;
  4. Advanced query tools for ad hoc/advanced data analysis.
  5. The road map and support plan for each component.

Instead of re-inventing wheels, our Video Telemetry solution leverages the following things from Application Insights suite:

  • AI SDKs reaching most client platforms;
  • HockeyApp acquisition and its future impact to SDKs on various mobile platforms;
  • AI infrastructure in Azure: data collection, processing, storage;
  • AI integration with Azure portal;
  • AI Content Pack for Power BI and its future improvements;
  • Advanced ad hoc query tool;
  • AI tight integration with Visual Studio 2015;
  • AI developer community;
  • AI Continuous Export of data to Azure blob storage.

In other words, our Video Telemetry is built on AI (Application Insights) + BI (Power BI with AI Content Pack) + KI (Kusto Integration).

With this solution, we should be able to answer all of the following questions (10 W’s):

  1. Who is the user?
  2. When did the user watch?
  3. Where is the user located?
  4. Which video did the user watch?
  5. Which quality levels did the user watch (either auto-adapt or manual switch)?
  6. Which audio stream/language did the user tune in?
  7. Which closed caption track/language did the user select?
  8. Which tech did the player use?
  9. Which device type/OS version did the user use?
  10. How long did the user watch for each video?

 

How Do I Use it?

In this section, we will focus on implementing such a video telemetry solution for a video player based on Azure Media Player API. You can find Azure Media API document and sample code from this link.

Keep in mind that this approach works for video players on most major platforms. The development languages are different but the approach stays the same. We use a browser player to illustrate the implementation. If you only care about desktop browser player and plan to use Azure Media Player to implement the browser player, you may consider using the offshoot of this work, a ready-to use Video Telemetry plugin of Azure Media Player. Details about this AMP plugin can be found here and in GitHub for the plugin.

The screenshots are for the actual Video Telemetry data for the Azure Media Test Player collected using this solution. Azure Media Test Player is a comprehensive test tool developed by the author to be used for building solutions on Azure Media Services. This page shows the technologies this test tool is built on. For highlights of the features supported by Azure Media Test Player, please see here.

Prerequisites

For the ease of the flow, we assume you already have the following ready:

  1. An Azure subscription;
  2. Visual Studio 2015;
  3. An ASP.NET MVC project (or any web project) containing the video player.
    1. The video player is based on Azure Media Player API.
    2. To be practical, it would be better your ASP.NET has user authentication included. If your player application does not need or have user authentication, everything still works but you will not be able to collect information on authenticated user or track each user’s activity.
    3. You have a couple of test videos you can play in the video player, regardless where the videos are hosted or whether the video is DRM protected.

Preparation in Azure

Application Insights enabled client applications send data to Azure for storage, processing and reporting. Therefore, as the first step, we need to create an Application Insights resource in Preview Azure Portal.

If your Azure Media Player application is an ASP.NET MVC application deployed as an Azure Web App, it may already have its own Application Insights resource. You can just use this resource and skip the following step for creating a new Application Insights resource.

To create a new Application Insights resource, in Azure Preview Portal, New -> Developer Services -> Application Insights

After an Application Insights resource is created. We need to get its Instrumentation Key which will be needed in the video player application.

Browse -> Application Insights, then select the Application Insights resource you have just created. Click on Settings icon on the top and select Properties on Settings and Diagnostics tab. Take a note of the instrumentation key for later use.

Add Application Insights to Player App

Next we need to enable Application Insights for your ASP.NET project. Follow steps listed here to add Application Insights into your ASP.NET project hosting the Azure Media Player code.

Custom Code for Video Telemetry

STEP 1: Download this javascript file for video telemetry and save it to the /Content folder in your project.

CAUTION: Please make sure to replace the instrumentation key in the configuration section of the javascript file with your own instrumentation key. Otherwise you will not be able to see your data.

Let’s reference the javascript file by adding the following line at the header portion of the video player page. If you use layout file for your ASP.NET app, you could choose to put it in the header portion of _Layout.cshtml file so that performance, availability and exception metric can all be collected as a side benefit for every page.

@Scripts.Render(“~/Content/appInsights.js”)

 

STEP 2: Capture Authenticated User Context:

If your ASP.NET application supports user authentication, you can add the following code right after the above line so that authenticated user context can be captured and correlated to video telemetry.

<script>

AppInsightsUtils.setAuthNUserContext(@System.Web.HttpContext.Current.Request.IsAuthenticated.ToString().ToLower(), “@System.Web.HttpContext.Current.User.Identity.Name”);

</script>

 

STEP 3: Track Video Player Events:

One of the most popular requirements of video telemetry is to collect statistics of user actions/player events such as play, pause, seek, bitrate adaptation/change, go full screen, change volume, playback finish, etc. Fortunately, Azure Media Player exposes 19 different events including all of these.

After amPlayer object is initialized, you can register a desired set or all of the events by calling the following method (you may comment out those events you don’t care).

AMPUtils.registerEvents(amPlayer, ampEventHandler);

where the function and ampEventHandler are defined below:

AMPUtils.registerEvents = function (amPlayer, ampEventHandler) {

var events = [amp.eventName.loadstart,

amp.eventName.durationchange,

amp.eventName.loadeddata,

amp.eventName.loadedmetadata,

amp.eventName.canplaythrough,

amp.eventName.waiting,

amp.eventName.play,

amp.eventName.playing,

amp.eventName.ended,

amp.eventName.seeking,

amp.eventName.seeked,

amp.eventName.pause,

amp.eventName.fullscreenchange,

amp.eventName.volumechange,

amp.eventName.error,

amp.eventName.timeupdate,

amp.eventName.playbackbitratechanged,

amp.eventName.downloadbitratechanged

];

 

for (var i = 0; i < events.length; i++) {

amPlayer.addEventListener(events[i], ampEventHandler);

}

}

function ampEventHandler(evt) {

//any event handling code here, if any

AppInsightsUtils.ampEventHandler4AppInsights(evt, amPlayer);

}

The last line in ampEventHandler function calls the custom javascript code in appInsights.js to track different kind of events with custom properties.

That is all you need to do to add video telemetry into your Azure Media Player web application. After some test runs, you can look into your telemetry data in any of the following ways. The latency is usually around 3 minutes. The latency of data arriving into Kusto is a bit longer.

 

Accessing Telemetry Data

Now you have added video telemetry capability into your player application, next we will look into three ways to access/visualize your telemetry data.

  1. Using Azure Portal;
  2. Using Power BI;
  3. Using a client, which provides ad hoc/advanced data query via a query language.

Of course, it is your choice. If you don’t have a need for ad hoc query or deeper data insights, there is no need to use the advanced query client.

Viewing Telemetry Data in Azure Portal

Log into Azure Preview portal. Browse -> Application Insights, then select the Application Insights resource you set up and used in previous section. You will see the pane containing general info of your resource and the Setting pane also appears on the right.

  • You can specify the time range of your interest via Time Range icon at the top;
  • You can go into Metric Explorer by clicking on Metric Explorer icon at the top.
  • You can invoke Settings and Diagnostics pane any time by clicking on Settings icon at the top;
  • In Settings and Diagnostics pane, you can click on Usage button to go to Usage data;

Please keep in mind of the following:

  1. You can add a chart;
  2. After adding a chart, you can customize it by selecting those attributes you are interested in.

As part of video telemetry, we collect data related to the following events:

  1. Playing;
  2. Pause;
  3. Go full screen;
  4. Change audio volume;
  5. Seek;
  6. Switch bitrate (either auto-adaptation or manual change);
  7. Interval (a timer event firing periodically at an interval you set, only during video playback).

The following table shows each event with corresponding custom properties and custom metrics collected together with the event:

Event Custom Property Custom Metric
Play
  • Video URL
  • Tech
Video duration
Pause Absolute time
Full screen change
Volume change
Seek
Playbackbitratechanged_[bitrate] Current bitrate
Ended
Interval
  • URL
  • Tech
  • Audio stream name
  • Caption track language
  • Current bitrate
  • Time watched (seconds)

 

Let’s first look at usage data. Click on Settings icon, then Usage button, the Usage Analytics pane shows up:

It contains a few charts.

  • If you click on the top area of each chart, a details pane shows up with a full display and other details for that chart;
  • If you click on a row in a chart, a details pane shows up with a full display and other details about that specific row of the chart.

For example, if you click on the top portion of “Top sessions by country” chart, it will display details of user/session versus country:

ByCountry

Next, let’s go into Metric Explorer by clicking on the Metric Explorer icon at the top of the main pane for the Application Insights resource.

If you click on interval row, the Search pane appears. If you click on each event, a detail pane shows up:

The “interval” details pane shows the custom properties and custom measurements. These data are collected by video telemetry at fixed time interval during video playback so that a full picture of user video playback can be captured. Of course, if this kind of details is not of interest to you, you can disable interval event.

 

Viewing Telemetry Data in Power BI

Application Insights team created an Application Insights Content Pack for Power BI. With this content pack, one can view video telemetry data in Power BI.

This document describes the steps to set up Application Insights Content Pack for Power BI.

After connecting to Application Insights resource, Power BI will now import your data, and in about another minute, you’ll see a dashboard like below for your telemetry data.

PowerBI

 

Advanced Data Query

What we can see in Azure portal and Power BI via Application Insights Content Pack are high level statistics, similar to “canned” queries in relational database world. Even though the data is there, the UI does not expose all. Sometimes, we woud like to do deeper and more detailed inspection of telemetry data to understand more. For this, there is an ad hoc query tool which is not yet available externally. Please reach out to the Microsoft account team to see how this tool can be leveraged.

UPDATE ON 9/6/2016:

An enhanced/revised version of Kusto client has been integrated into Azure Preview Portal. You can just use Azure portal instead of Kusto client for ad hoc data query.

After opening an Application Insights app in Azure Preview Portal, click “Analytics” button to open the Kusto Query Language tool.

For example, you can query a specific user’s activities using a query like below:

customEvents

| where user_AuthenticatedId == “johndoe@outlook.com”

//| where telemetryType == “event”

| where timestamp >= now(-30d)

| where timestamp <= now()

| extend custMedaurements=parsejson(customMeasurements)

| extend custDimensions = parsejson(customDimensions)

//| extend typeDimensions = parsejson(typeDimensions)

| extend playTime = custMedaurements[“playTime”]

| extend bitrate = custMedaurements[“bitrate”]

//| extend eventType = typeDimensions[“name”]

| extend srcUrl = custDimensions[“srcUrl”]

| extend tech = custDimensions[“tech”]

| extend audio = customDimensions[“audio”]

| extend cc = custDimensions[“cc”]

//| where eventType ==”interval”

| project timestamp, user_AuthenticatedId, client_IP, client_City, client_StateOrProvince,srcUrl, tech, audio, cc, playTime, bitrate, session_Id, client_Model, client_OS

| sort by timestamp desc

Continuous Export of Telemetry Data

If you want to export Video Telemetry data for long term retention or custom analysis, you can perform Continuous Export on Application Insights data. The events you see in the Application Insights portal can be exported to storage in Microsoft Azure in JSON format. Continuous Export is available in the free trial period and on the Standard and Premium pricing plans.

For detailed steps of setting up Continuous Export of your Application Insights, please see here.

Continuous Export will export telemetry data into Azure blob storage as JSON files.

Azure Stream Analytics

Azure Stream Analytics (ASA) makes it easy to set up real-time analytic computations on data streaming from devices, sensors, web sites, applications and infrastructure systems. It supports a powerful high-level SQL-like language that dramatically simplifies the logic needed to visualize, alert, or act on incoming events in near real-time.

You could use Stream Analytics to pipe your data to SQL DB, Table Storage, Event Hub or Power BI. This opens up many possibilities of how you process, analyze and view your data. For example, the following two document explain how to set up Stream Analytics to pipe data to Power BI:

 

Summary

We have presented an all-Azure solution for Video Telemetry/user analytics based on AI + BI + KI. This solution can reach clients on all major platforms: iOS, Android, Windows 10 and all major browsers.

We provided detailed steps and source code for such a solution for Azure Media Player client.

We also presented two ways to look at Video Telemetry Data.

Have fun with Video Telemetry!

Your feedback is very much appreciated. Please direct your questions/feedback to me at willzhan@microsoft.com.

ACKNOWLEDGMENT: The author is grateful to Dale Koetke of Microsoft Application Insights Team for his help throughout this effort. Special acknowledgment goes to Amit Rajput and Vishal Sood of Azure Media Services Team for their encouragement and support.

Comments (0)