Run cloud-based load tests using your own machines (a.k.a. Bring your own subscription)

When you run a cloud-based load test, the load testing service automatically provisions the necessary machines (load agents) for generating the load to your application. Once the load test run has completed, these resources are torn down. This works well for the most part for a large set of customers. However, some customers want to be able to run load tests using their own machines – be it virtual machines in Azure that they provision in their own subscription or other machines, virtual or physical, that may be living on-premises.

This blog looks at the two primary scenarios where such a configuration may be useful. Before we begin, let’s get familiar with some terms that we will use through the rest of the blog, for simplicity.

  • Auto-provisioned machines: These are load generating machines that are automatically provisioned by the CLT service when a load test run request is received and are also automatically torn down when the load test run has completed execution. When these machines are used, you are charged VUMs, as applicable for your load test run.
  • Self-provisioned machines: These are load generating machines that you can provision on your own (in your Azure subscription or on-premises). These machines can be configured to register themselves against your VSTS account and then they can run a cloud-based load test. This is the focus of our discussion in this blog.
  • Cloud-load agent: This is the agent that can work with the CLT service. This agent will be installed when you use self-provisioned machines and needs to be configured for your VSTS account. Once configured, it can then be used for running a load test. The cloud-load agent is NOT the same as the Test Controller/Test Agent that you may have used earlier for running load tests or automated tests. Towards the end of the blog, we will see the differences.

Now, let’s look at the scenarios:

  • More control over agent machines – Sometimes, you may need more control over the agent machines – for e.g., to install custom software that is used during a load test run. While simple configuration is easily achieved using deployment items and setup scripts on auto-provisioned agents, if you are installing some bulky software or doing some time consuming operation as part of the setup, you may want to do that only once and reuse the machines over and over again, in order to save time and effort. Since the auto-provisioned agents are also torn down automatically, using your own machines instead that you can setup and tear down at will can help.
  • Testing private apps / apps behind the firewall – The basic requirement of the cloud-load testing service is that the application endpoint be public or reachable from the cloud. Often times, that isn’t the case. The app that you want to load test may live entirely on-premises behind the firewall or in a private VNet in Azure or you may be developing new features that will only become publicly accessible when it’s released. How do you then load test such an app? If you could provision agents in the same network as your app (so that they can reach the app) and have them work with CLT, you could load test your app easily.

The question that then remains is “How”:

  • Using machines on-premises: You can provision as many machines as you need on-premises and run a powershell script that will install the cloud-load agents and configure these machines against your VSTS account. For more details and the powershell script, refer this blog post. The agents communicate with the CLT service only using HTTP(S), so you don’t need to open any other ports.
  • Using virtual machines in Azure: While you can certainly adopt the same approach as (1) where you provision the machines and then run the powershell script to install and configure the load agents, we have simplified this process using ARM templates. You need to specify just a few inputs such as your VSTS account, a PAT token for authentication and the number of agent machines you need and you will be on your way to creating and configuring the machines in a single shot. The machines will be provisioned in your Azure subscription (hence the “Bring your own subscription”) and you will have complete control over these machines. We have provided 2 ARM templates in the Azure quick-start templates repository on github that you can use. Let’s see which template to use when:
  • Simple template with dynamic IP option: This template will provision the number of machines you specify and assign them dynamic IPs. With this configuration, the application will still need to be publicly reachable. You can install any software you may need after the machines are provisioned and ready or you can customize the ARM template to install the necessary software as part of the provisioning process itself.
  • Simple template with static IP option: This template will provision the number of machines you specify and assign them static IPs. The machines will get the same IPs even after you shut them down and restart at a later point in time. With this configuration, you can allow traffic from these known IPs through the firewall to reach an application behind the firewall. The agents communicate with the CLT service only using HTTP(S), so you don’t need to open any other ports.
  • VNet based ARM template: This template will provision the number of machines you specify in a specific VNet that you have already setup in Azure. This VNet would be where your private app is also hosted and thus the load agents have a line of sight to the app and can thus reach your app.

For more information on using these ARM templates, please refer this blog post.

If you have been using Visual Studio load testing for some time now, you may already be familiar with Test Controller/Test Agent (TC/TA) that can be used to run load tests. And you may be wondering how that is different than the cloud-load agent usage described above. Here are the differences:

  • Cloud-load agent does not need a controller. The CLT service acts as a lightweight controller instead. Test Agent on  the other hand needs a separate Test Controller.
  • Cloud-load agent on self-provisioned machines uses the CLT service to store results and benefits from any enhancements we make to the CLT service. For e.g., you can view the load test report in the browser. The Test Controller/Test Agent uses SQL Server to store results.
  • Cloud-load agent uses HTTP(S) for communicating with the CLT service and is quite resilient to network glitches whereas the communication between Test Controller and Test Agent uses .NET remoting, which makes it quite susceptible to network glitches. The Test Agent is also a lot more chatty than the cloud-load agent.

Hope you enjoyed this one! Try it out and reach us at vsoloadtest@microsoft.com with any questions or feedback.

FAQ

  • How do I run JMeter tests using the self-provisioned option?

>> The cloud-load agent has been designed to run a JMeter test too. Enabling the experience to use self-provisioned agents for a JMeter load test instead of the default auto-provisioned path that exists today is on the backlog.

  • What is the cost of using self-provisioned agents ? Will I be charged VUMs?

>> If you use self-provisioned agents on-premises, you bear the hardware cost, of course. If you use self-provisioned agents in Azure, the machines will be provisioned in your Azure subscription and you will be charged the applicable cost by Azure, based on the number of machines and the duration for which these VMs are running. CLT will not levy any VUM charges in this scenario.

  • Can I use the self-provisioned option when running load tests in a CI pipeline?

>> Yes, certainly. You can use the existing Azure RG tasks from the build / RM catalog with either of the ARM templates to provision, start or stop machines. The CLT tasks have also been updated so that you can specify whether to use the auto-provisioned agents or self-provisioned agents. A build template that will help you with this configuration is in the works and will be made available soon. Stay tuned!

  • How do I automate the process of provisioning machines and shutting them down?

>> You can do so with using a build definition. In this context, think of the build (CI pipeline) as an automation orchestrator rather than a system that does build. We will have a blog post outlining this one, stay tuned.

7