Speed up cloud-load test execution by retaining resources for quick consecutive runs

Validating application’s performance by running a load test typically follows a test->fix->test loop, often repeated several times. After you have run an initial load test and made some changes (either on the app side or test side by fixing issues or tweaking configurations), you want to quickly validate if that works and gives you the desired results or if you need to go back in the test->fix->test loop. What we have found is that often times, the “test” part of such a loop is typically a short duration one and the faster it goes, the better the productivity. When cloud-based load testing service runs a load test, it automatically provisions resources (load agents) in Azure. Depending on the # of agent machines required for the load test run, the resource provisioning may get done quickly or take a while (sometimes as much as 20 minutes) and at the end of the run, these machines are recycled. This means that often times you cannot do a test->fix->test loop fast enough, as each “test” may experience a good amount of delay in starting, waiting for machines to get ready.

To eliminate this issue with delays, we have added the capability to retain resources from one iteration of the test that can be reused for subsequent iterations, so that you can get through the desired number of test iterations in the amount of time that you have at hand. Let us see how this feature works.

How to turn on resource retention

When you use Visual Studio to run load tests, this feature can be accessed using the newly added Resources Retention setting in the load test’s active run settings. This is available in Visual Studio 2015 Update 3. If you are using prior versions of Visual Studio, you can add a context parameter called ResourcesRetentionTimeInMinutes in your load test to access this and assign the desired retention time value.

ResourceRetentionSettings

Retain resources setting – Visual Studio 2015 Update 3

Resource retention in older VS versions

Retain resources setting – older VS versions

 

 

 

 

 

 

 

 

 

In the above screenshots, the setting is to retain resources for 100 minutes. So, from the time that your load test run finishes, for the next 100 minutes, the machines will be kept alive, so that you can trigger another run without any delay. You can retain resources for a maximum of 4 hours (240 minutes).

If you do not happen to trigger another run in that time, the retention duration will expire at the end of the 100 minutes and machines will be reclaimed. After this expiration, if you trigger another run, it will acquire machines afresh and the amount of time taken will depend on how fast Azure is able to service that request.

You can also choose to extend the retention time further if you need it. For that, you simply adjust the retention time value to the desired duration in another run and the retention will get extended for the specified duration.

If you have multiple active runs, each with the retention setting, each set of resources will be retained and can be reused for subsequent runs. The following graphic shows an example of how this works and at what time will the resources expire.

Example of load test runs using retention setting

Example of load test runs using retention setting

 

FAQ

  1. Is there a charge for this feature?

Ans: Yes, there is a small charge applied when you use this feature. In addition to the usual VUMs that your load test consumes, a surcharge of 5 VUMs per core will be applied for the duration of the run. So if you use retain 20 cores for 10 minutes, a 1000 VUMs (5*20*10) additional will be charged. You will see this information appear in the status messages of your load test run.

2. If I have retained resources for a given duration and decided not to use them further, can I release them earlier?

Ans: No, this is not possible at this time. Once retained, the resources will be alive for the duration you retained them. They will be released only after the retention duration expires.

3. I am running Apache JMeter tests from the VSTS portal. How can I use this feature?

Ans: At this time, the feature is only available for load test runs triggered using Visual Studio. We will be bringing this ability to the web portal soon, so that you can use this with tests such as Apache JMeter tests that are triggered from the portal.

4. I need some more information. Who can I reach out to?

Ans: You can reach us at vsoloadtest@microsoft.com for any load test related queries.