VSTS 2010 Load Test Feature: Saving Test Logs

Now that VSTS 2010 beta 1 is available I and other Load Test development and test team members will be writing a series of blog posts highlighting the new Web and Load testing features. Ed Glas has a blog post that lists the new features here. You can download the beta here.

This post describes in detail the item in Ed’s list “Log entire test result on test failure …”:

With VSTS 2010, Load Test users now have the option of capturing and saving the entire result of individual tests run within the load test such as a failed Web test or a failed unit test. This feature makes it easier to debug problems that occur when running tests within a load test that do not occur when running the same tests outside the context of a load test.

The New Load Test Wizard exposes a new option “Save Log on Test Failure” which defaults to True:

Also, in the Load Test editor, the Load Test Run Setting’s property sheet includes this option as well as two other related properties in the new “Logging” category:

You can see the description of the new “Save Log Frequency for Completed Tests” property above. The “Maximum Test Logs” setting controls the maximum number of Test Logs that are saved in the load test results database for a single load test. You can increase this value from the default value of 200, but we aware that the test logs could potentially take up a considerable amount of space in the database especially for Web tests that contain many large requests and/or responses since they include the entire body of each Web test request and response (the exception to this is that details for dependent requests are not saved in order to save space).

For example, to see how this works with unit tests in a load test, suppose I have the two following unit test methods:

[TestMethod]
public void UnitTestThatLogsAndPasses()
{
Debug.WriteLine("This line written using Debug.WriteLine()");
Trace.WriteLine("This line written using Trace.WriteLine()");
Console.WriteLine("This line written using Console.WriteLine()");
System.Threading.Thread.Sleep(1000);
}

[TestMethod]
public void UnitTestThatLogsAndFails()
{
Debug.WriteLine("This line written using Debug.WriteLine()");
Trace.WriteLine("This line written using Trace.WriteLine()");
Console.WriteLine("This line written using Console.WriteLine()");
System.Threading.Thread.Sleep(500);
Assert.Fail("This test fails for demo purposes");
}

Then I include these unit tests in a load test with the passing test running 90% of the time:

The Test Logs are not viewable while the load test is running, but can be viewed once the load test has completed and you go to the post-run Load Test Analyzer by responding Yes to this prompt at the end of the load test:

(You can also bring up the post-run Load Test Analyzer for a load test run from the “Open and Manage Load Test Results” dialog available on the toolbar in the Load Test Editor.)

This should bring you to the Summary page in the Load Test Analyzer; if you then click on the Errors link, the Errors table is displayed and includes links to the Test Logs in the “Details” column (the right-most column – you may need to scroll right to see it):

Clicking on one of the “Test log” links in this table opens a new Window in Visual Studio to view that unit test result which looks just like viewing the result of a unit test run outside of a load test:

In addition to links to the Test Logs in the Errors table, there is also a new “Test Details” table in the Load Test Analyzer that also may contains links to Test Logs:

In the load test result shown above, it was a Web test that failed, so clicking on the “Test Log” link opens a new Web Test result viewer window in Visual Studio; you can see the details of requests that failed, but you can also browse around the Web test result to see the details of previous requests that may have been considered successful but perhaps caused later requests to fail because they were missing expected content (and there was not an appropriate validation rule).  

As with the unit test, the Web test result viewer is the same as when the Web test is run outside of a load test:

As a slight aside to this topic, the new Test Details table is available in the post-run load test analyzer as long as you have set the value for the Load Test Run Setting’s property “Timing Details Storage” to “All Individual Details”. This is true even if you disable the capturing of Test Logs by setting “Save Log on Test Failure” to False and leaving ““Save Log Frequency for Completed Tests” at the default value of 0 (which means never capture test logs for successful tests). In this case, the Test Details table would look like this:

When “Timing Details Storage” is set to “All Individual Details” (which is now the default value for this property for load tests created with VSTS 2010), the data shown in the table above is stored in the load test results database for all tests run within the load test (excluding the warm-up and cool-down periods). The Test Details table in the Load Test Analyzer will only display up to 1000 of these rows at a time. When the table is first loaded, it is sorted in chronological order (that is by ascending values of the Start Time column) as shown above. However, you can choose a different sort column and toggle the sort order by clicking on the column headers. This results in a new query to the load test database so that you see the top 1000 rows matching the sort column and order. For example, you can choose to sort by the slowest values in the Test Time column to see the 1000 slowest individual tests within the load test.