A/B Test All the Things: React Native Experiments with App Center

Visual Studio App Center

Visual Studio App Center helps developers ship better apps faster, but it also helps you build features your users will love.

How? We designed App Center to be API-first and fully modular. This means that when you integrate App Center services with your existing toolchain, you have the flexibility to apply them to other parts of your development process in new and innovative ways, such as seeing which features your users like best. Today, I’m highlighting one of my favorite creative uses: creating an A/B testing framework for React Native apps with the App Center services CodePush, Push, and Analytics.

As a refresher, A/B testing is a popular tool to validate design and implementation decisions whereby you deploy two application versions of your app and see which one users prefer (depending on your app, this may be more engagement, more purchases, or uploading more content). You iterate quickly based on your results, and then ship the best version to all users.

Let’s get started!

The Basics

CodePush delivers app content updates to React Native (and Apache Cordova) apps over the air (OTA), so you push bug fixes and feature updates directly to active users.

By publishing multiple revisions of individual apps (defined as separate ‘Deployments’ in CodePush), you’ll have multiple versions of your app ready for deployment to various user segments. The core version is your base app content that’s delivered with the app in its initial deployment (through an app store, or using App Center’s Distribute capabilities), then you’ll use your published ‘A’ and ‘B’ revisions to validate design decisions, or test new features on a subset of your app users.

Configuring CodePush to A/B Test

First, use the CLI to create new CodePush deployments for your app. By default, App Center creates Production and Staging deployments for your app.

• The Production deployment hosts the default version of the app, the one that everyone gets when they download the app from an app store or when a beta version is deployed using App Center Distribute service.
• The Staging deployment is home to your test versions. For our purposes, you’ll need one or more additional deployments (your “A” and “B” variants).

For my demo, I created RevisionA and RevisionB deployments. To do this:

• Use the CodePush CLI code-push deployment add command, passing in the name of the app, plus the name of the deployment you’d like to create.
• For example, if my project is called MyApp, to create a RevisionA deployment, I’d use: code-push deployment add MyApp RevisionA

When you’re done, the deployments list in App Center will list the additional deployments.

Figure 1: Visual Studio App Center CodePush Deployments

Next you’ll create the A and B versions of your app (usually in different version control branches), then publish each separately to your RevisionA and RevisionB deployments.

Use the code-push release-react command.

For my MyApp example, I’d use: code-push release-react MyApp android --d RevisionA

Pushing Deployments

With your A and B versions completed and ready to deploy, you’re ready to use App Center’s Push service to trigger app deployments and get your “experimental” apps into your users’ hands.

• To do this, send a push notification that contains a custom data object referencing the revision version’s Deployment ID and deploy to your defined user segments.
• To obtain the deployment ID, you’ll use the CodePush CLI: code-push deployment ls APP_NAME -k, for example: code-push deployment ls MyApp -k

Push notification processing information lives in your app.js file; when your app receives the notification, it retrieves the deployment ID and automatically pulls the correct revision from CodePush.

Visual Studio App Center Push includes an Audience feature, allowing you to send notifications to a subset of your users, using one or more app or device properties.

Figure 2: Defining a Push Audience

When you’re done A/B testing and want to reset your app to its default content, simply push the Deployment ID for the Production deployment to the same audience. You can also switch all users to the winning app version by pushing the Deployment ID for the winning version to all users.

Understanding the Results

Once you’ve deployed your “A” and “B” to your user segments, App Center’s Crashes and Analytics capabilities make it easy to see how your app versions behave in the wild, and how your users engage with each version.

• Crashes gives you detailed, immediate insight into when and why your crashes, as well as how many users are affected.
• Analytics helps you track users’ activity in app versions, with all information visualized in App Center Analytics dashboard to make it easy to see trends or identify gaps.

With the App Center Analytics SDK, you define custom events (depending on your app, this can range from identifying which menu choices are made to tracing a user’s path through the app) in your app to understand how users use different features, where they’re successful, and where they get stuck, so you make informed decisions about which version works better, based on your app’s goals.

Getting Started

This is just one example of how to improve your development processes; using tools and services to run experiments, validate hypothesis, and get real data that helps make decisions and improve your apps (and keep your users coming back).

If you haven’t done so already, get started for free with App Center and see what it can do for you. Reach out to me on Twitter at @johnwargo to let me know what you think!

When I presented this at ReactiveConf 2017, the demo gods smiled upon me, so my presentation went without a hitch. Check out my presentation video and slides, and dig into the sample project source code and step-by-step implementation guide here.

Visual Studio App Center

Follow