Scheduling Load Test Execution

After publishing  Load Testing from different geographies with Cloud-based load testing I have have had a couple of people ask how do you schedule Load Tests. 

Depending on what type of load testing you are doing there are two ways to do this. 

  1. For on-premises Load Test you will need to use the Command line (see directions for doing this at the bottom of this post)
  2. For Cloud Load Testing we make this simple with the new Build System

While you could use the command line for both on-premises and Cloud-based Load Test (see http://blogs.msdn.com/b/visualstudioalm/archive/2015/08/24/cloud-load-test-support-in-mstest-exe-command-line-and-xaml-builds.aspx) the new Build System for Cloud-based Load Test makes this much easier.

Scheduling a Cloud-based Load Test as part of the a Build

In addition to being much easier, the new build system in Visual Studio Online as a scheduling engine also has the added benefit of being able to connect your load test into an entire workflow and able to kick off release pipelines with approvals and end to end reporting.   To get started:

1.  Create a new Build

Navigate to the Build hub and create a new build

image

In the Build templates select empty…as we will only need the Load Test Tasks

image

2.  Create a Service End Point

Cloud-based Load Testing execute from a remote agent and must have access and “know” which VSO Account to publish the results back to. This is done through Service Endpoints. To create a Service Endpoint for Load Test you must go to control panel add a Generic service set to the VSO Account you want the load test to run from.

REMEMBER THIS REQUIRES A PAT OR ALTERNATE CREDENTIALS.

I always seem to forget this!  Directions for setting alternate credentials and creating a PAT can be found here:

http://roadtoalm.com/2015/07/22/using-personal-access-tokens-to-access-visual-studio-online/

image

3.  Setup the Load Test Tasks

Add both the Quick Load Test Task and Load Test Task

image

For the Quick Web Performance Test simply up the parameters for your quick performance test:

image

For the regular Load Test Task you will have to create a Load Test Using Visual Studio Enterprise and Check it in to the repo we are building

Note the Testing settings file is typically in the root of the SLN.

image

3.  Scheduling the Build

Under the triggers tab of the build definition select Triggers and select the “Scheduled” option. 

image

Scheduling an on-premises Load Test 

Sean Lumley’s post below will show you how to run a web test or load test from the command-line instead of from within Visual Studio.

That post and the use of the Windows Task Scheduler

image

Most of the time you will want to run you web or load test from within visual studio, but you can run them from a command-line. Maybe you have a set of 5 web tests that you want to run once a day. You could configure a scheduled task to run a batch file which contained ran the 5 web tests. The command line executable that is used to run tests is called mstest.exe and is installed when you install either VS Team Suite or VS Team Test SKU. Here is a MSDN help link for mstest: http://msdn.microsoft.com/en-us/library/ms182486.aspx

The easiest way to get started with mstest is to start a visual studio command prompt. Go to Start Menu -> All programs -> Microsoft Visual Studio v9.0 -> Visual Studio Tools -> Visual Studio 9.0 Command Prompt. This will launch a command window which has mstest.exe in its path. So type mstest /? This will show each of the available options. Let’s run through a few options of how to run tests.

1) Running a single WebTest

First let’s run a simple web test. To do this, we just need to specify the /TestContainer argument. A .webtest file or .loadtest file is considered a test container in the same way that a dll which contains unit tests is a test container. So if we have a web test called WebTest1.webtest, you would use the following command to run the web test:

mstest /TestContainer:WebTest1.webtest

This will produce output similar to the following:

Loading WebTest1.webtest…

Starting execution…

Results Top Level Tests
——- —————
Passed webtest1.webtest

1/1 test(s) Passed

Summary
——-
Test Run Completed.

Passed 1
———
Total 1

Results file: C:\Documents and Settings\slumley\My Documents\Visual Studio 10\Projects\TestProject3\TestProject3\TestResults\slumley_SLUMLEYTCM 2008-12-22 13_38_14.trx

2) Running Multiple WebTests

Now let’s execute 2 web tests from one call to mstest. You can specify multiple test container arguments on one command-line. So say we wanted to execute WebTest1.webtest and WebTest2.webtest. You would use the following command-line:

mstest /TestContainer:WebTest1.webtest /TestContainer:WebTest2.webtest

This will produce output similar to the following:

Loading WebTest1.webtest…

Loading WebTest2.webtest…

Starting execution…

Results Top Level Tests

——- —————

Passed webtest1.webtest

Passed webtest2.webtest

2/2 test(s) Passed

Summary

——-

Test Run Completed.

Passed 2

———

Total 2

Results file: C:\Documents and Settings\slumley\My Documents\Visual Studio 10\P

rojects\TestProject3\TestProject3\TestResults\slumley_SLUMLEYTCM 2008-12-22 13_4

3_04.trx

3) Specifying deployment items

One of the big differences with running from the command line is that you are not within a visual studio project. When you run a web test from within Visual Studio, we will try to determine what needs to be deployed with the web test in order for it to run. For example, maybe you have written a custom validation rule or a extraction rule. When you run the test from within visual studio, we will look at the references of the test project and try to figure out what needs to be deployed.

When you are running from the command-line, you will need to be very explicit about what you deploy. So if you have a dll that needs to be deployed for a test to run, you will need to inform mstest. The way you do this is by specifying /runconfig parameter. One of the sections of a run config is deployment items. Use visual studio to create a run config. Then to run a test with a runconfig, you would have a command-line similar to the following:

Mstest /TestContainer:WebTest1.webtest /RunConfig:NewRunConfig.testrunconfig

4) Executing a test on a controller/agent setup

When in VS, to run a web or load test on a controller/agent setup you need to use a test run configuration. You would open the run config, specify to run the test remotely and specify a controller name. It works the same way from the command-line. You will need to create a run config that has the correct settings. Then you run it the same way as you do when specifying deployment items:

Mstest /TestContainer:WebTest1.webtest /RunConfig:NewRunConfig.testrunconfig

5) Running a coded web test

Running a coded web test is similar to running a unit test. You will set the /testcontainer argument to the dll which contains the coded test.

mstest /TestContainer:TestProject1.dll

One thing you will notice when running a test this way, is that mstest will execute all of the tests within the dll.

6) Executing a specific coded test in a dll

So if you want to execute just one test within a dll, you would need to use the /test argument. For example, to run WebTest1Coded, you would use the following command line:

mstest /TestContainer:TestProject1.dll /Test:WebTest1Coded

If you wanted to execute 2 tests, you can specify multiple /Test arguments. For example:

mstest /TestContainer:TestProject1.dll /Test:WebTest1Coded /Test:WebTest2Coded

7) Executing a load test

This is the same as executing a web test except you need to specify the .loadtest file

mstest /TestContainer:LoadTest1.loadtest

When the test completes, you can either open the trx file to get to the load test results or you can use the manage load test results dialog form within VS. Here is a blog on that: http://blogs.msdn.com/slumley/pages/managing-load-test-results.aspx

8) Specifying the results file name

You will notice that the results file(trx file) gets a unique name which contains user, machine and a timestamp. If you want to specify the name of the results file and where it is generated, you can use the /resultsfile parameter. For example, to create a results file called MyResults.trx and have it saved to c:\results, I would do the following:

mstest /TestContainer:WebTest1.webtest /resultsfile:c:\results\MyResults.trx

Hopefully this post shows you how to make use of the mstest tool.

0