Analyzing Test Results in MTM - Part II

Introduction

In the previous blog post Analyzing Test Results in MTM– Part 1 we have seen how to find the run and a snapshot of various test run analysis features. In this blog post we will look at Analyzing a test failures using Test case result view and how to get a final report. In the sample run provided in the blog post there were 11 inconclusive results and 5 failures. We have analyzed 11 inconclusive results in the previous post using Test run view. Mow let’s analyze the 5 failures.

 

Test case result view

There are 5 failures in the run and let’s analyze each one of them using the above features. Since these are genuine failures in the run we need to open each test result to get more information on the failure from the logs. As usual user can double-click on the test result or Click “View Results� Button to view the test case results. The below analysis screen comes up where it displays all the data that the user is looking for. Browse through the below screenshot and take a look at the data displayed in each of these fields.

Test_Case_Result_View

In the above screenshot you can see related items Combo Box which will let you open the Test plan, Test run and the test case that the test result belongs to.

Related_Items

Attachments

Based on the data collectors turned on in the test settings, user will find all the log files associated to the run in the attachment section. If you have enabled video collector, you shall even look at what exactly happened during the test execution. Users can also write custom data collectors to gather specific information that they wanted from the test machine which will help them analyze the results. Note that if the attachments are huge and coming from multiple agents there may a time lag for all the attachments to get into TFS. In these cases you will attachments with 0KB of size, this means the files are getting uploaded to the server.

Attachments

Even the TRX file has been saved here. Opening the TRX file will open it VS IDE. User can investigate the failure and then update the analysis information in the results individually.

TRX

Test result History

Users can view the past test result history in the same view where they can find if the same issue has happened earlier. \

Test_Result_History

Update analysis information

Based on the investigation users can update the test result with analysis information. The below screenshot shows updated fields. Like this we shall analyze all the other 4 failures.

 

Update_Analysis_Information

 

When is analysis complete?

Notice that unless there are no failures with failure type value “None� or Resolution type of “None� or “Needs Investigation�, the analysis is not complete. Since all this analysis information is preserved, it is meaningful to analyze all run failures and update all the fields with appropriate details so that if the same issue happens in future it can easily be auto-analyzed. Also all the analysis data gets moved to warehouse where users can get historical reports about the past test runs and can get cool and meaning reports using excel reporting feature.

 

Final Analysis chart

After the analysis of the tests the final results looks like this. The user get to see some pie charts which gives a snapshot of the results based on failed test and analysis of those tests

Final_Analysis_Chart

 

Summary

With this blog post Analyzing Test Results in MTM series is complete. In this blog we have seen following in detail

- Analyzing a test failures using Test case result view

- Analysis complete indicator and final report

Click here to get to Analyzing Test Results in MTM - Part I

 

Author – Venkatesh Sargunesan

Test Lead – CodedUITest