A/B Testing with Azure Websites


In the context of websites, A/B testing, or Testing In Production, is a technique whereby different users of a website will be randomly presented with different versions of the site. This is commonly used to test proposed enhancements with a limited number of real users, allowing the website owner to measure the effectiveness before the enhancements are rolled out more broadly.

A/B Testing is not a new concept, and it’s conceptually straightforward – however implementing it effectively in your own websites can be very difficult without specialised tooling. However if you’re hosting your sites on Microsoft Azure Websites, it’s incredibly easy to set this up. At the time of writing (November 2014), Azure Websites Testing in Production is fully implemented in the New Portal, but documentation is very limited – hence my decision to create this article. 

The Basics

The key to Azure Websites Testing in Production is that you deploy two completely different copies of your website, and tell Azure which percentage of users should see each version. I’ll provide a more detailed walkthrough below, but the basic steps are:

  1. Create a new Azure Website and select a Standard hosting plan
  2. Create a new Deployment Slot for each additional version of your site you want to deploy (up to 5)
  3. Build two or more different versions of your website using the language and tools of your choice
  4. Deploy the “primary” version of your site to your “primary” Azure Website, and then deploy each additional version to a different named Deployment Slot
  5. Use the Azure Portal to configure the Testing in Production feature, specifying which percentage of users should go to the primary site and each deployment slot when they browse to your site’s URL
  6. Use Application Insights to measure the effectiveness of each version of the site.

That’s really all there is to it. Let’s look at these steps in more detail.

Walkthrough

In this walkthrough I’ll be using the new Azure Portal. Some of the steps can be done in the older Azure Management Portal, but the Testing In Production settings can only be defined in the new portal.

1. Create a new Website

To create a new website from the portal, click New, choose Website, and then choose a name, location and hosting plan. You’ll need to choose one of the Standard hosting plans in order to get access to multiple deployment slots. I chose to call my site azuretip, so its URL will be http://azuretip.azurewebsites.net. For a real-world site, you would of course map your own DNS name to the site.

image

2. Create Deployment Slots

Next we’ll use the portal to create additional deployment slots. A deployment slot is basically a completely new website, with no shared code and configuration with other slots in the same Website. However (as with Staging and Production slots in Azure Cloud Services) you can “swap” slots to implement staged deployments (but that’s not what we’re doing here!).

In the blade for your main website, find the “Deployment Slots” section, click it and create a new named slot. I called mine beta, but since it’s a slot of azuretip it results in a new site with a URL http://azuretip-beta.azurewebsites.net.

image 

3. Build your Websites

Next you’ll need to build two or more versions of your website. This can be done in any language supported by Azure Websites (e.g. ASP.NET, PHP, Java, Python) and whatever tools you want. There’s no need for your multiple versions to share anything in common, although in practice you’ll most likely be deploying different versions or branches of the same codebase. In my case, I have a fantastic web store selling the coolest products at crazy low prices. However sales have been stagnating, and I have a theory that by changing to a dark theme I can entice users to buy a lot more. Here are my two sites:

image image

4. Deploy your Websites

With my two websites built, I’m now ready to push them to Azure. I’m going to use Visual Studio’s Publish dialog, but there are plenty of other ways you can do this. Whatever approach you use, make sure you deploy the “primary” version to the primary website (in my case, azuretip) and the additional versions in the correct named slots (in my case, beta which displays as azuretip(beta) in the publish dialog).

image

5. Configure Testing in Production

We now have two versions of our site deployed to Azure, but people will have to browse to different URLs (http://azuretip.azurewebsites.net and http://azuretip-beta.azurewebsites.net) to see the different sites. However in order to run a successful A/B test, users should randomly be presented with just one of the sites. This is what the Azure Websites Testing in Production feature does. Not only will it randomly choose a version based on the percentages you configure, each user will remain on that version throughout their visit as long as the same ARR Affinity cookie is presented.

To configure Testing In Production, open the new portal, browse to your website, and find the “Testing in Production” section on the blade. From there, you can quickly configure what percentage of users should go to which deployment slot when they browse to the primary URL (i.e. http://azuretip.azurewebsites.net in my case).

image

Testing that everything is working can be a little tricky due to the affinity cookie, but if you open up the home URL in multiple browsers or in “in private” mode or similar, hopefully you’ll (eventually) see examples of different versions of the site:

 image image

6. Measure Effectiveness

So with multiple versions of the site being served to different users from the same URL, we’re done, right? Well, kind of – but in order to be useful, A/B testing needs to be run as a controlled experiment where you monitor how people interact with your site and make a determination as to which version is better. This also requires that you know how to measure effectiveness, which will vary depending on your site. For example on an ecommerce site you’ll likely focus on the number of transactions, whereas on a social network you may care about number of posts or number of advertising impressions.

Regardless of how you choose to measure effectiveness, you’ll need data. There are lots of ways you can get data about site usage, but one extremely simple but powerful approach is to use Application Insights. This is enabled by default on new projects created with Visual Studio 2013 Update 3, and results in site usage and performance data getting pushed to Azure and made visible in the portal.

My sample sites don’t have enough pages (or real users) to yield anything truly “insightful”, but the dashboard should still give you an idea of how this could help measure the effectiveness of your various deployed versions. If you’re going to use page views as one of your metrics, make sure you factor in the Testing in Production percentages before comparing results.

image

Conclusion

A/B Testing helps website owners make better decisions on how to evolve their site, by providing real-world data from a limited set of users that can be compared against the experiences of a larger control group. If you deploy your site to Azure Websites, setting up a A/B testing is incredible simple, so why not give it a try? And if you’re not using Azure, you can sign up for a free trial.

Comments (5)
  1. Galin Iliev says:

    For testing that everything works you can delete the TipMix cookie or force navigation to slot by appending query string ?x-ms-routing-name={slotName} to force slot navigation or x-ms-routing-name=self to navigate to base production slot.

  2. Thanks Galin – great TIP! (Pun somewhat intended… 😉

  3. Galin Iliev says:

    🙂 Great article. Thanks for covering it.

  4. Guruprasad says:

    Hi,

    Is there a way to configure user selection based on identity than percentage?

    We would like to route certain users(what we call as preview users) to dev/stage version while rest of the traffic goes to production code. URL approach works for initial load, but this wouldn't help with AJAX calls we have unless we modify that code to use the URL parameter.

    Thanks

  5. Pavan Prem says:

    if we have deployment slots associated with each web app, say a staging slot for each web app, how do those slots factor into pricing?

Comments are closed.

Skip to main content