Using Application Insights to Improve Mobile Beta Testing

Editor’s note: The following post was written by Windows Platform Development MVP Matt Lacey as part of our Technical Tuesday series.

There are lots of different types of testing. In this article I’m going to refer to what is often called “beta testing”. That is testing that is performed by a small number of people outside of the core development team of an app. These “beta testers”, from now on just referred to as testers, may be part of the organization that is creating the app or members of the public. The purpose of beta testing is to gain further confirmation that the app behaves as intended in real world environments and gain feedback from users before the app is released to a wider audience.

Beta testing should not be seen as an alternative to testing done internally and there is rarely a good reason to give testers a version of an app that has lots of bugs in it. Beta testing instead extends the breadth of environments that the app can be tested in. This might include different hardware devices, more configurations or localizations that cannot be tested internally due to limits on skills or resources.

When it comes to beta testing, unfortunately, many developers release the app to their testers and then just sit back and hope for feedback. In many scenarios just hoping for feedback isn’t sufficient and so to ensure you're getting both the most feedback possible and the test coverage you would like it is important to be proactive about capturing feedback and monitoring what the testers are doing.

Application Insights (https://azure.microsoft.com/services/application-insights/) provides a way of satisfying both of these goals. It allows us to receive details of any problems that testers may encounter and also to see what they are actually testing.

One of the most important reasons for beta testing is to ensure that the app works as intended. One major way an app may not work as intended is if there are unhandled exceptions that cause the app to crash.

By default, Application Insights will record all unhandled exceptions for you. Even if you have your own way of recording exceptions and allowing testers to email those exception details to you it can be valuable to also record them with Application Insights. Doing so will mean you can be sure of getting all the details, as long as they have a connection to the internet, and will also put all the details in one place making them easier to analyze or spot trends than if you have to extract the details from the bodies of emails.

Even with these benefits it is still valuable to allow testers to submit exception reports via email. It is not always possible to reproduce an error from an exception report or stack trace and so being able to ask the person what they were doing when an error occurred can be invaluable. Additionally, it is also often beneficial to the relationship with your testers to be able to tell them when an exception they encountered has been fixed. Being able to reply to their email is a great way to do this as it is personal and shows that you value the time and effort spent to report the issue. 
 

The above image shows a number of crashes. There is definitely an issue with range checking that should be investigated and fixed but in reality there are more things that may happen when using an app that we’d like to prevent and which aren’t crashing exceptions.
In some scenarios we may trap situations which would otherwise cause an error in code and report them directly to the user. While not directly an error, this may be something we want to know about as it may indicate an issue with the usability of an app.
The simplest way to track scenarios like this is with custom events.
var tc = new TelemetryClient();
tc.TrackEvent("ClientValidationFailed-Terms");

The above code snippet shows how a user failing to check the “I accept the Terms and Conditions of usage” box when trying to register may be recorded. In this scenario, client validation failure means the user is stopped from doing something when they tried. The image below shows the client validation failures that were recorded in the registration page of one app during testing. 2
 

This is from the event Filter as it is how we can clearly see the number of times each event was recorded.

The high number of validation failures where the password was not entered the same way twice, may indicate that the registration page may need some improvements. It is however, also important to consider these figures in context. What these figures don’t show is how many people were able to successfully register. Without this knowledge the above figures could be either small, and so not worth considering, or large and so indicative of a major potential problem. There are two ways this could be addressed. First an event could be recorded when validation is successfully passed. This would provide a data point in the Application Insights dashboard that is easy to compare. Alternatively, we could compare with figures gathered elsewhere. In this instance we could look at how many people successfully registered during the testing period by counting the number of new accounts created on the backend. With knowledge of this figure we could then make informed decisions about the consequences of the number of client validation failures. There is no hard and fast rule about whether it is best to record successful actions as Application Insights events or not. It will depend on individual circumstances and whether the data is also available separately or not.

In addition to knowing when things go wrong it’s important to know what is actually being tested. It is not uncommon for some people to volunteer to test an application but never actually get round to it or never look beyond the first few pages in an app. For this reason, just having had people volunteer to test an application does not provide any guarantee of the volume or extensiveness of that testing. We want to be confident that our app has been exposed to and used in a wide variety of environments before we make it public but just having people say they will test it is not the same as them actually doing so. If we want to be confident that testers are doing what we want it is necessary to track their activity. The most basic statistic for tracking what testers are doing is to record page views.

Page views are recorded by default when Application Insights are added to a project and provide a simple metric that can indicate how much testing each part of an app has received based on the number of views.

While a simple metric it can easily be skewed as, depending on the navigation structure of an app, the default behavior may not be what you want. The default behavior is to record a “view” every time a person navigates to a page. This is not just new pages but also includes navigating back to a page viewed previously. Consider a simple, two page, master-detail app scenario where the user launches the app on the main page, clicks something to navigate to the detail page and then presses back to return to the main page, this would result in two page views for the main page and one for the detail page. Depending on your requirements this may not be what you’re after. Fortunately it’s easy to change.
The first step is to disable the default behavior.

In a Universal Windows Platform (UWP) application you disable the automatic recording of page views by removing (commenting out or deleting) the PageViewTelemetryModule from the ApplicationInsights.config file.

In a Windows 8.1 or Windows Phone 8.1 project you disable the automatic recording of page views by not specifying it as a collector when initializing the telemetry collection.
WindowsAppInitializer.InitializeAsync(
    "INSTRUMENTATION-KEY-GOES-HERE",
    WindowsCollectors.Metadata | WindowsCollectors.Session | WindowsCollectors.UnhandledException);

If we want to record page views ourselves it is a very simple task to do so. The code below shows the simplest way of doing so and also prevents views being counted multiple times as the user navigates away from and then back to the same page.
protected override void OnNavigatedTo(NavigationEventArgs e)
{
    base.OnNavigatedTo(e);

    if (e.NavigationMode != NavigationMode.Back)
    {
        var tc = new TelemetryClient();
        tc.TrackPageView(nameof(MainPage));
    }
}

With page view tracking in place we can see which pages in the app the testers are viewing. 
 

The aim here is to check that all parts of the app are being adequately used and therefore tested. From the above chart we can see that very few of our testers have visited the Login page. There may be many reasons for this but it is probably a page that we would want to see tested further before releasing the app to the public.

For most pages, in most apps, we expect the person using the app to do more than just navigate to the page. As such, we want to make sure our testers are actually using the app in the ways intended and not just navigating around the different pages. To know what the testers are actually doing in the app it is useful to track specific events. The exact events that will be appropriate to track will vary depending on the specific app but may include activities such as sharing; exporting or importing data; altering settings; or searching.

With some of these events it can also be useful to record additional related properties. Application Insights includes a way to easily do this by providing them in a dictionary, as below.
private void Search(string searchTerm)
{
    if (string.IsNullOrWhiteSpace(searchTerm)) return;

    var tc = new TelemetryClient();
    var properties = new Dictionary<string, string>
                             { { "term", searchTerm } };
    tc.TrackEvent("Search", properties);

    ...
}

An app may also have pages that display different things at different times. This may be a generic page that behaves differently or with different layouts depending on the data being displayed. If we have a single page that could display a user, a product, or a picture it may not be very useful to know the total number of times that page was visited. It is normally much more useful to know how many times it is used to display each of the different items.

Additionally if an app is localized to support multiple languages it can be important to check that all parts of the app are being tested in different languages. Because Application Insights doesn’t make accessing the device language easy to query and filter, I’ve found it useful to combine this with the page name and any other key information affecting how the page is displayed as a custom name which, when tracked, is much easier to query on the portal.
protected override void OnNavigatedTo(NavigationEventArgs e)
{
    var type = ((Item)e.Parameter).TypeName;

    var pageName = string.Format(
        "{0}_{1}_{2}",
        nameof(Detail),
        type,
        CultureInfo.CurrentUICulture.Name);

    var tc = new TelemetryClient();
    tc.TrackPageView(pageName);
}

The telemetry gathered in this way makes it easy to see how the app is being used and the languages that the testers are using. The data below is the list of page views filtered to the Detail page. From here it is clear that while the page has been used to view each of the item types many times it will probably be beneficial to have some more testing of the page by testers who have their device in German, assuming the app is localized to German. 
 

In addition to testing with devices in each language an app is localized for it is also beneficial to ensure that our testers are representative of actual users. Consider a group of testers who are all new to an app that provides access to data collected elsewhere. If the testers are new to the service they may not have collected much data previously and therefore need to load and display and so find the app is very fast. However, a long time user of the service may have lots of data and find the app very slow to load.

Or consider a social networking app that allows the user to retrieve a list of all their friends. With only a small number of friends the app may be very responsive but may start running slower as they make more friends in the app.

For many apps, different people will have very different experiences based on their settings and usage. It is important to monitor this to ensure that your testers cover a wide variety of users (or potential users if the app is brand new) and to ensure that all of them have a good experience with the app. As a starting point for doing this you can collect some simple metrics like those below.
var sw = new Stopwatch();
sw.Start();
List<string> friendNames = await GetFriendNamesAsync();
sw.Stop();

var tc = new TelemetryClient();
var measurements = new Dictionary<string, double>
{
    { "friendCount", friendNames.Count },
    { "loadTime", sw.ElapsedMilliseconds }
};
tc.TrackEvent("LoadedFriendNames", metrics: measurements);

Below we can see an example of such data in the portal. By filtering to the specific event of interest, it is easy to quickly go through the details and ensure a representative spread of values in the data and that there are no significantly large load times. 
 

It is important to note that if you start to record more information about your testers as part of your event tracking be sure to avoid any privacy issues around the use and storage of personal data by not capturing anything personally identifiable.

In this article I’ve shown some of the reasons it’s important to monitor what your beta testers are doing with the app and how they’re using it. Application Insights provides a number of ways we can record this information to help us ensure that the people testing our apps are actually using/testing all of it and doing so in ways that reflects what actual users will do. With this information we can be confident of releasing a higher quality app that more users will love.

About the author
Matt is a mobile development consultant and also runs the Windows Apps London user group. Follow him @mrlacey.