Visual Studio Online Load Test Troubleshooting Guide

Trevor Hancock

March 14th, 20140 0

Visual Studio Online Load Test Troubleshooting Guide
Last Updated June 24

This guide has moved to FAQ for Cloud-Based Load Testing and will be maintained there going forward. Please refer to that page for the most recent content.

GENERAL

**1. How do I know the IP range of Test Agents used for my load run?****
**All the test agent machines are located in East US Azure data center. IP range of East US datacenter can be found here. Currently, you cannot determine the IP range of the Test Agent machines used for only your load test run.

**2. Where are the load test results stored after downloading them to the client machine from cloud?****
**By default these are stored in the local database (SQL Express). Using SQL express is fine for storing the test results from trial run. For longer term use, kindly use SQL Server to store the load test results by following these instructions. SQL Server also provide better performance than SQL Express.

**3. Does the machine where the agent is hosted for my run get shared with any other run?
**No. The virtual machine where agents for a particular run are hosted does not host agents for another run.

**4. Can I run load tests with Visual Studio Online using mstest?
**There is no support for running load test using mstest currently.

**5. Where are the load testing agents located for my run? Can I host the load testing agents at my preferred location?
**Currently all the load testing agents will be located in Azure – East US data center. No, you can’t change the data center where the Agents are hosted.

**6. What are VUserMinutes and how do I know how many VUserMinutes are consumed by the run I am initiating on the service?
**VUserMinutes = (Maximum User Load for the run) * (Duration of the run in minutes).

Note: The minimum values used for calculating VUserMinutes are 25 VUsers and 1 minute. If your run has a smaller number of VUsers OR it is running for less minutes, then that particular value will be rounded off to the minimum value. E.g. if your load test is going to run with 100 VUsers for 30 seconds, then instead of calculating with 30 seconds, VUserMinutes will be calculated with 1 minute: VUserMinutes will be 100(VUser) * 1 (minute) = 100. The same rule applies to VUser count calculation. Also, if your run duration is 5 min and 15 seconds, then the run duration will be rounded off to the next integer value i.e. 6 minutes here.

There will be a deduction of VUserMinutes from your account for any runs that are in the “Completed” state as mentioned above.
There will be no deduction of VUserMinutes from your account for any runs that are in an “Error” state.
There will be a deduction of VUserMinutes from your account for any runs that are in an “Aborted” state. VUserMinutes will be calculated as mentioned above with the following changes.
User load will still be the Maximum User load as mentioned in your load test.
Run duration will be the actual time your run was executed in the cloud.
Minimum of 250 VUserMinutes will be deducted (including warm-up period)

You can see how many VUserMinutes have already been used by your account by checking the main account page (e.g. https://XYZ.visualstudio.com) and looking at the “Load Testing value in the “Resource Usage” section in the lower right:

** 7. ****What are the current resource limits of running my load tests using Visual Studio Online? This includes changing the limits for VUserMinutes, Maximum cores allowed in the run and Minimum VUsers per agent core.
**These policies are per Visual Studio Online account. A trial account has the following resources free per month: maximum of 15,000 VUserMinutes per month. If you need more virtual user minutes for your load testing, have your account owner get additional resources for your Visual Studio Online account.

a. For the trial VSOnline account, if the free resource limits are exhausted then your load run will get aborted and you will get a status message like this:

“This run exceeds the maximum allowed usage for this month. The current usage for your account for this month (including runs in progress) is 8000 and the maximum allowed usage is 15000. To learn more about usage limits and how to modify them, refer to http://go.microsoft.com/fwlink/?LinkId=303976”

b. Maximum number of cores for the load test agents for your run is 20 cores.

c. Minimum VUsers per agent core is 25. This value is calculated by dividing ‘Maximum User Load’ by ‘Total Agent cores’ in your load run. You need to have minimum of 25 VUsers per agent core in your run.

If your load test has a requirement of a greater number of agent cores OR lesser number of ‘VUsers per agent core’ please contact Microsoft support for help.

CONFIGURING TESTS

**1. How do I datadrive the load test?
**Currently to datadrive your load test execution in Cloud, you can only use .csv, .xls. Using SQL is not supported currently. You can read more about it here.

**2. How long do I have to wait until I can run my load test in the cloud after creating a VSO account?
**It can take 5 sec to 3 hrs. to get the permissions to run the load test in the cloud. If you had previously created the VSO account, you may be able to run the load test right away.

**3. How do I calculate the number of agents which ****should be used for my run?
**The number of agents to be used by your run is totally based on your tests. If you get errors when you run your test, you may need to increase the number of agents. Make sure the ‘Agent Count’ property in Run Settings is set correctly. This is a new property to set the no. of agent cores you want to use for running your tests on Visual Studio Online. By default it is set to 0.

If Agent Count (Total Cores) is set to 0, then the total cores allocated will be based on the number of Vusers in your test.

Total Agent cores = (Number of Vusers / 250 ). That means that for each 250 VUsers, your test gets an extra core with max of 20 cores. Number of test agents allocated will be similar as mentioned below.

If Agent Count (Total Cores) is set to 1, then your test run will be allocated one agent with one core.

If Agent Count (Total Cores) is set to 2 or more, agents will always be allocated with 2 cores. E.g. if the Agent Count (Total Cores) is 4, then 2 agents each with 2 cores will be allocated to your run. Also, it will never allocate agents of size 1. So, for Agent Count (Total Cores) = 3, then 2 agents each with 2 cores will be allocated to your run.

Also, there is a limit to the number of virtual-users per core which a test run can have. The ratio of virtual-users/core needs to be a minimum of 25 for the load test to be executed on the service. E.g. if you were executing the test with 6 VUsers and 2 Agent cores, this makes the ratio of virtual-users/core as 6/2 = 3 which is less than 25. Hence, you will not be able to execute such tests in the cloud. Although, for the test runs which are running with the default value of Agent Core Count (which is zero), this check will not be applicable.

You will see the following status message, when this validation check fails for your test: “A load test cannot be executed with less than 25 virtual-users/core for one run. Reduce the agent count or increase the virtual users in your load test and run the test again”

The number of test agents required for your load test are dependent on the kind of test mixes (web performance test OR unit test) that you are including as part of the load test. If you have only web performance tests then the recommendation is to have 250-1000 VUsers per 2 agent cores. For unit tests in the load test, we don’t provide any recommendations. Also, you can test if the number of test agents will be sufficient by executing a smaller duration load run first.

**4. How do I change the number of Agents used in ****my run?
**To change the number of agents, you need to edit the Agent Count (Total Cores) property in Run Settings as shown below. By default the value is kept as ‘0’, it will still allocate one agent with one core for your run.

**5. How do I install certificates/software on load ****testing Agents?
**You can use deployment options and a Setup Script in test settings. You can add the .exe or other files to the Deployment window which you want to deploy on the Agent and using the Setup script you can install them on agents.

All the items deployed on the Agents are copied to a directory on the Agent. The location of the directory can be accessed by %DeploymentDirectory% in the setup and cleanup script. E.g. If I want to install WebDeploy on the agent machine I should add WebDeploy_x64_en-US.msi to Deployment window. Setup.cmd will look like %DeploymentDirectory%WebDeploy_x64_en-US.msi /passive

RUNNING AND MONITORING TESTS

**1. How do I check Load Testing Service status?
**To check the status of the service please refer to the service status at the top of our Visual Studio Online support portal and our service BLOG. You can also subscribe to alerts for service status by following this post in our support forum.

**2. What are the possible load test run states ****when running in the cloud with Visual Studio Online?
**Your run can be in multiple states while it is running in the cloud. The major steps involved and what they mean for your run are listed below:

In-Progress: The test run is in progress on the cloud.

Completed: The test run has completed successfully on the cloud.

Aborted: The test run is stopped as the user forcefully stopped the run by hitting the stop button. This state can also appear if there are issues related to your load test. E.g. if there are issues in your test scripts, etc.

Error: The test execution is stopped due to an error occurring at the service. E.g. there are infrastructural issues in the service and it is unable to execute your test further. This has nothing to do with test scripts or a problem due to your load test.

3. How should I view Test logs after downloading the test results locally.
There is an issue at our end and we are in middle of fixing the issue (MAY 2014). Please use this workaround in the meantime:

Open Notepad in admin mode.
Open the devenv.exe.config file in it.
Change the following highlighted property to 14 from 11.

<dependentAssembly>
               <assemblyIdentity name="Microsoft.VisualStudio.QualityTools.LoadTest" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
               <bindingRedirect oldVersion="8.0.0.0-14.0.0.0" newVersion="12.0.0.0"/>
           </dependentAssembly>

Open Visual Studio again and try opening the Test logs. It should work.

TROUBLESHOOTING TESTS

**1. What should I do to record a web performance test with IE 11?
**If the web test recorder is not getting enabled while recording your web test with IE 11 then please follow the link to resolve the issue.

**2. What should I do if Visual Studio ****freezes/hangs when I try running the test in cloud?
**Follow the post here for a solution.

**3. How to view errors and warnings occurring ****during my run on Visual Studio Online?
**There can be two types of errors that occur while your load test is running in the cloud.

a. Status messages: Relate to load test execution in the cloud. E.g. like what all agents have gone down, connection to the results DB was lost, etc.
b. Test errors: Relate to each web performance test or unit test execution on Test Agents

Both of these can be viewed in the “Details” tab as shown below.

**4. I am not able to import the downloaded test ****results and it is showing an error message that the connection’s current state ****is closed. What should I do?
**You can set the amount of time a connection waits to time out by using the ConnectTimeout or Connection Timeout keywords in the connection string. A value of 0 indicates no limit, and should be avoided in a ConnectionString because an attempt to connect waits indefinitely.

**5. Unable to use > 250 VUsers or plug-ins even ****if I have VS Ultimate full license using Sign-In Experience
**As a workaround please take the VS Ultimate Product Key from your MSDN subscription and use the “Change my Product License” option on the option on the Product Information page. You would need to do this on every machine which you want to use for load testing. Visit this site to get the product key(s).

6. How do I check Load Testing? I’ve noticed user ****code failing to execute if it depends on the test names? Are test names ****changed when run against the service?
Test names in Load tests run against service are converted to lower case, hence any string match done on test name by user code should ignore case or convert test names to lower case.

**7. How do I enable ****client-side logs to help troubleshoot issues with Hosted Load Tests?
**Use a text editor to edit devenv.exe.config. This file is generally located in “C:Program Files (x86)Microsoft Visual Studio 12.0Common7IDE”.

a. Add the following inside the … section:

Where XXX can be any of the following:

“all” – logs all messages
“off” – stops logging of all messages
“critical” – only logs Critical messages
“error” – only logs Error/Critical messages
“warning” – Default logging level, logs Error/Critical/Warning messages
“information” – only logs Error/Critical/Warning/Info messages
“verbose” – only logs Error/Critical/Warning/Info/Verbose messages

b. Add the following <system.diagnostics>…</system.diagnostics> section to the bottom of the devenv.exe.config file, just above tag. You can specify the path for the log file by changing the initializeData value.

<system.diagnostics><br /> <trace autoflush="true" indentsize="4"><br /> <listeners><br /> <add name="myListener" type="System.Diagnostics.TextWriterTraceListener" initializeData="d:VSTestHost.log"/><br /> </listeners><br /> </trace><br /> <br /> <switches><br /> <!-- You must use integral values for "value": 0 = off, 1 = error, 2 = warn, 3 = info, 4 = verbose.--><br /> <add name="EqtTraceLevel" value="4" /><br /> </switches><br /> </system.diagnostics>

c. Restart VS 2013 and reproduce the issue. You can then review the log file or share it with support.