Running Cross Browser Automated Testings in Visual Studio Team Services

Editor’s note: The following post was written by Visual Studio and Development Technologies MVP Mike Douglas as part of our Technical Tuesday series with support from his technical editor Jeff Bramwell, who is an MVP in the same category.  

Many teams have implemented automated Continuous Integration (CI) and deployment processes into production, and some teams have started to include infrastructure provisioning as part this process.  But one area that is often the last to be incorporated is automated testing. If your deployment process only takes ten minutes,  but takes 5 days to manually run regression tests -  then your total deployment time is really five days and 10 minutes.

So why does testing take so long? And why does it have to be the long pole in the tent, and prevent your team from realizing true DevOps benefits?

The most common reason tests takes so long is because they are manual, which keeps your DevOps efficiency in the gutter. So, we need to incorporate automated testing into our testing process.

Are unit tests enough?

Sadly, no. I have several concerns about relying heavily on unit tests. One challenge is that they are hard to map to functional requirements. This is why I put a lot more effort into providing automated functional tests. These can be mapped directly towards your acceptance criteria, which should cover the detailed requirements.

So should all automated tests be UI tests?  

While UI tests are ultimately the best way to test the UI as users will use it, UI tests are notoriously slow and fragile. But this doesn’t mean you should skip them. UI tests provide the ability to validate things that can't be done at any other level.

Today, applications are expected to run on more platforms, devices, and browsers than ever before. But to do this, you need an unlimited timeline and budget (and if you do, I’d like to be on your project)

The reality is, that the cost is too high to create and maintain tests that work across that many combinations of configuration. I recommend that looking at the website logs to see where the majority of your customers are using. Typically, two or three options stand out and that is where you should initially focus your effort.

Let's assume that the latest version of IE and Chrome are the top two browsers used by your clients.  We want to create a set of UI Tests to validate functional requirements. We want to be able to use the same tests for both browsers and write the tests so they are maintainable  - and that small changes to the UI don't require changes for  every test. We typically use Selenium tests with the Page Object Pattern or LogiGear’s TestArchitect to accomplish both of these goals.

In this example I will use Selenium tests, found in this GitHub repository. (There is still some refactoring to do on the example, so don't beat me up too bad about the code). Once we have the tests created, running these tests on a developer's or QA Engineer’s machine is not a viable option, as they can take hours to run on a single machine and require full control over the desktop while the tests are running. 

Rather, running the tests in a build and release system such as Visual Studio Team Services (VSTS) provides a number of key benefits - including scheduling these to run in a nightly batch, using a pool of machines to distribute the tests so the test run can complete sooner, leveraging Azure to spin up/shutdown the machines so they are only running and costing us during the test run, and being able to pass in test configurations so the same tests can be executed for each configuration.

I will showcase most of these benefits in the example below. If you don't have access to a VSTS account, you can create a free account for up to 5 developers and unlimited repos. Along with the free VSTS account, you get 240 free build minutes per month.  This is more than enough for some simple usage. The following will also work with TFS 2015.  However, based on which TFS 2015 update you currently have, the task names and options may vary slightly.

VSTS supports executing tests as part of the build or release. These utilize the same web-based cross-platform system. While if I were doing this as part of the software delivery pipeline I would run these tests from a release,  in this example I am simply going to build the test project and run the tests - so executing this all from a build is sufficient.  

Below is the customized build definition. The highlighted tasks are the ones I added. I removed the test task because these UI tests run as part of the Run Functional Tests tasks and I don’t want the UI tests to run as part of the unit tests. If you have both, make sure you use different naming conventions within the projects and test tasks to ensure the tests you want to run are executed.

The highlighted steps show that we are going to start the test client machines, copy the required testing DLLs to the test client machines, dynamically deploy the test agent and execute the tests for both Chrome and IE, and then shutdown the VMs.

I’ll walk through the highlights of these steps and then include the information on how to configure the test client machines:

One of the benefits of using Azure, is that you only have to pay for what you use. Azure VMs are required to run the automated UI tests and you can keep these turned off when the tests are not running. To do this, use the Azure Resource Group Deployment (see images below) tasks for both the start and shutdown VM actions. These tasks will perform the actions against all of the VMs in the resource group by default. You will have to configure your Azure RM Subscription service connection. The Resource Group is the container that contains one or more test client machines that will be used to execute the tests.

The next step is to copy the test DLLs from the build agent to the target machine(s). Choose the source path used for the publish task and a destination folder for the target machines. Ensure that the Test Certificate checkbox is checked. This is one of several tasks that is required to use WinRM properly on the test client machines. See the Configuring Test Client Machines section below for more details. Credentials are required to remotely authenticate to the machine so it can copy the files. The same Resource Group you used above should be used in this step, as well.

With TFS 2015 and VSTS, the test agent can be dynamically installed as part of the build/release process.  This has a number of benefits, including that the test machine will always be in the correct state and it is easy to spin up dynamic test client machines. The Deploy Test Agent task doesn’t use the Resource Group to specify the group machines. Instead, pass in a group of machines as a comma delimited string. There will also be a couple other tasks that will require this, so use a variable to make this easier to manage.

Ensure that Interactive Process is checked so the tests can run UI tests on the desktop and again, check the Test Certificate box.

You might have noticed there are two tasks for deploying the agent and running the tests. The deploy task needs to be run before each test run to ensure the test agent is ready.  We will focus more on the two Run Functional Tests tasks next as these are primary focus of the article.

The Run Functional Tests tasks will each run through the full set of tests specified by either assembly or test suite selections.  In this case, the task will run through all of the tests in the assembly.  Often I will use the test plan and test suite option to be able to select the particular tests that I have associated to test cases, but this isn’t the focus of this article. Be sure the same test client machines are chosen to run the tests you used for the deploy test agent task.

In both of these cases I used the $(TestClientMachines) variable to manage this.  The Test Drop Location should be in the same folder as the target of the Azure copy task above.  The run settings file provides a number of features.  For the sake of this article, the settings file allows us to pass in parameters to the tests from task.  This provides the key benefit of making our tests reusable across environments and cross browsers.  In this task, the Override Test Run Parameters is set to browser=chrome and the IE task (not shown) has it set to browser=ie.

The last required option  you should enable is the Continue on Error, because you’ll want the other tasks to continue to run even if some of the tests fail.   The Reporting Options are not required but provide the ability to view the results by browser type so it is high recommended to set these too.  The Test Run Title allows the tests to be viewed by browser type in the build results screen (see the last screenshot) and the Test Configurations associates the test run to a Configuration in the Test Hub.  The format is a little tricky. In my case, I’m just using default configurations and the ID for Chrome is 26. You can see the ID when you view the configurations in Test > Configurations.

Configuring Test Client Machines

Azure provides an ideal environment for running UI Tests. If you have a Visual Studio Subscription you can activate your Azure benefits and receive up to $150 / month credits. If you don’t have a Visual Studio Subscription, you can instead utilize your corporate account or start a trial.

You can create an Azure virtual machine and if you keep it off, you are only charged a few dollars per month for the storage. When the virtual machine is turned on, you are charged for the compute cost which can be a lot more. This is why you only want to have the virtual machines on while the tests are running. The creation of Azure virtual machines can be accomplished through the Azure Portal or automated with PowerShell or ARM Templates. If you have never created a VM in Azure, I would recommend doing this first through the Azure Portal. I chose a Windows Server 2016 Datacenter server with appropriate resources, such as an A2 with 3.5gb RAM. Also, when you create it be sure to choose an appropriate name for the resource group. As mentioned above, this name will be used for several tasks.

VSTS can configure everything required to communicate within the virtual machine via WinRM. However, by default the RDP port is the only port open. To allow incoming WinRM traffic, add an inbound rule for WinRM.  You can do this during the VM creation or afterwards.

The second setting is to pick a DNS name. By default, the VM is created with a public IP but not a public DNS name. In the Overview settings click on the Public IP > Configuration and enter the name of the server.

Once the machine is created and these settings have been configured, you need to configure IE and Chrome to be able run the tests consistently. The following are a few settings I change on new machines.  For other tests you may need to make some additional changes.

IE

Most modern windows server OSes have IE Enhanced Security configured by default. From the server manager > Local Server, select the On link for IE Enhanced Security Configuration and select Off for both settings.

Open IE and choose to Use recommend security, privacy, and compatibility settings and press OK.

If you try to run the tests now, you may receive an error like this one, that states the IE Protected Mode settings are not the same for all zones.

Go to the Internet Options in IE and either change them all to protected or not protected. I chose to disable the option under Internet and Restricted Sites. Even though this is less secure, this should be ok because the only websites accessed should be through the automated tests.

Chrome 

Obviously Chrome doesn’t come installed on the machine, so download and run the installer from the web. For this sample test, all of the default settings are fine. Optionally you can change some other settings, like allowing popups.

Viewing the results

With everything configured and your build or release completes, the test results are displayed so you can quickly identify which tests passed and failed.

Conclusion

Being able to run functional cross browser automated UI tests can provide your teams with the confidence to deploy your applications with quality, reduce the time it takes to complete regression testing, and improve your time to market. VSTS and Azure provide the tools and platform for executing these tests efficiently with the ability to infinitely scale the number of test client machines. Working through the example in this article should help you to get started.

Mike Douglas is a Solution Consultant at Deliveron Consulting Services.  He specializes in working with development teams to implement DevOps and Application Lifecycle Management (ALM) solutions, eliminating traditional silos between development, testing, project management, and operations in order to establish cohesive processes - from idea to production. He is also a Microsoft Visual Studio and Development Technologies MVP and a Microsoft ALM Ranger. Follow him on Twitter @mikedouglasdev