Analyzing Test Results in MTM – Part I

Introduction

CodedUITest can be executed in many ways, from VS IDE, MSTest.exe and from Microsoft Test Manager. If the user has run the tests from VS IDE or MSTest, he can publish the results to TCM Server using the publish button in the test results viewer in VS IDE. Once he publishes the results to TCM Server, he can use Microsoft Test Manager (MTM) to analyze the results. In this blog post we will look into how to analyze a test run using MTM.

MTM UI Workflow

To understand the MTM UI workflow and the features please take a look at Activity Center and Testing vs Planning blogs. These blogs talk about how MTM UI is laid out and how users can navigate to various views. Please read these blogs before proceeding further if you are new to MTM.

Test Run View

All Test runs are stored under Testing Center -> Test -> Analyze Test Runs in MTM. By default this view shows all the automated test runs that are executed in the last 14 days.

Customize the view

User can customize this view by clicking on “Show manual runs� to show manual test runs executed from Test Runner or change the start date range value to expand or narrow their search. Below screenshot shows how to customize the Test run View.

User can customize this view by clicking on “Show manual runs� or change the start date range value

Identify the run

First let’s look at how to find the run that the user just started or the run just completed. User can find the run based on run build number, Owner and creation date. Like all the views in MTM user can filter column values and group runs using columns names. User can even group runs based on column. Once the user find the test run user can click on Open button on the top or double-click on the run to view the results. Below screenshot shows how to filter the test runs, owner column is user for grouping and build number is used for filtering.

Run Details

The run details that you are seeing in the below screenshot shows all the aspects of the run like test settings, Controller, lab environment, run type and owner. It also shows the charts on the analysis information and users can also open the associated test case if it needs to be updated. Now let’s analyse this run.

Analysis features

In the above sample run there are 11 inconclusive results which the test has thrown based on the configuration it is run on. These tests were run on this config by mistake so let’s analyze these tests with “Failure type� as “Known issue� and “Resolution� as “Configuration issue�. Notice that you can select multiple items and analyze them together.

Failure Type

Users can choose one of the 5 options as failure types based on the reason of the failure and “None� is the default value. Failure type can be a Regression if the same issue was found and fixed earlier and it has regressed again. If the bug is not a known issue not it is a regression, we shall mark the failure as New issue. In this case this is known issue where the test should not have been run, so let’s select “known issue�

Resolution

Users can choose one of the 5 options as resolution and “None� is the default value. In the below example we have set the resolution as “Configuration issue� where this problem happens in this particular config. If we have analyzed the failure as a code defect we shall mark the resolution as Product issue or if the failure happens because of a test defect, then we shall mark the resolution as test issue. We shall mark the resolution as “Needs investigation� if we are just browsing through the failures and analyzing the simple and easy ones first and look at the hard ones later, users can select this option .

Reset test result

The button highlighted in yellow in the below snapshot will reset the test result and the related test will run again on the controller agent. If there was any mass failures, users can reset all the failures so these tests can rerun again. MTM will reset the test results to active so that test controller can pick them up. Users can even reset a passed test case, if there is a need.

Notes

“Notes� is where users can details on the investigation and why the test failed. This data will be very handy when the user has found a similar issue in later builds or when the user files a bug this details will be moved to the bug for the developer to take a look.

Create or link test results with bugs

Once the failures are analyzed, users can file bugs or link existing bugs to the test results by clicking on “Create bug� or “Link to Bug� buttons.

Summary

In this blog on Analyzing Test Results in MTM – Part 1 we saw the following in detail

· Find the run from “Analyze Test runs� view

· Features of Test run analysis

· Analyze some of the failures from Test run view itself

In the part-II blog we shall see the following

· Analyzing a test failures using Test case result view

· Analysis complete indicator and final report

Author – Venkatesh Sargunesan

Test Lead – CodedUITest