IntelliTrace and Microsoft Test Manager
One of our main goals with IntelliTrace was to help close down the “no repro” gap between developers and testers. It’s frustrating and a waste of time and resources to have testers and developers going back and forth with testers reporting issues and developers unable to repro and investigate those issues in their development environment. IntelliTrace provides an obvious bridge of this gap with the ability for testers to collect iTrace files on their test machines and then hand those same files off to developers to debug in their usual Visual Studio environment. But this scenario only works if we make it as easy and seamless as possible both for testers to collect these files and for the developers to be able to open them associated with the proper bugs. To this end we’ve worked hard on IntelliTrace integration with both the new Microsoft Test Manager and Team Foundation Server work item tracking. Note that both IntelliTrace and Test Manager will be available for the first time with the Visual Studio 2010 wave of products (IntelliTrace and MTM are available in Visual Studio 2010 Ultimate and MTM is available in the Visual Studio Test Elements SKU).
Microsoft Test Manager
Before we get deeper into the integration of IntelliTrace and Microsoft Test Manager it’s best to have a little basic knowledge about what Test Manager is and how it can help your testing efforts. Test Manager is a new application that runs separate from Visual Studio and is used for creating, managing and running tests associated with a TFS project. It covers a broad spectrum of automated, manual, unit and load tests and includes test plan management to help coordinate testing efforts.
During test execution Test Manager can help to collect various data items such as screenshots, desktop videos and comments to be associated with each test run. Then, using TFS, this data can be persisted and attached to any bug that is filed off of a test failure. This is where IntelliTrace comes into the picture as IntelliTrace iTrace files are one of the items that can be collected during a test run and attached to a bug. So with a minimum of effort a manual tester can capture a file chock full of debugger info and have it automatically attached to the bug that they are currently filing. Then when the developer opens the bug in TFS they are able to open up the iTrace file in their IDE and start debugging into the exact run that caused the tester to file the bug. It’s important to note here (especially if you’ve not read one of the more general articles on IntelliTrace) that the IntelliTrace log just contains a specific set of debugging data that is only collected at specific points during execution. The developer is not actually getting a fully debuggable / executable look at the program that was running on the tester’s machine. That being said, the data provided from an iTrace file, especially one in which collection has been tuned to common problem issues in the code being tested, can be very usefully in either guiding the start of a debugging investigation or in diagnosing the issue totally.
Collecting IntelliTrace data with Microsoft Test Manager
Microsoft Test Manager is a big, big new part of Microsoft’s testing strategy going forward. So as someone not even on the Test team I’m not going to do it the disservice of trying to cover all of its functionality. If you are interested in more on Test Manager I’d suggest starting with the MSDN docs or the Team Test blog, for this article I’m just going to be covering how IntelliTrace integrates with it and will be assuming that you know the basics of connecting MTM to your TFS solution and have a few tests and test plans available.
For setting up IntelliTrace data were going to start by navigating to the Test Settings tab in the Lab Manager section of MTM. Then either create a new set of test settings or open up an existing test settings item. From the main test settings page click on the “Data and Diagnostics” category over on the left side to navigate to the screen shown below.
From this screen you can just click the box for IntelliTrace to turn automatic IntelliTrace collection on for the current set of test settings. If you click the Customize button to the right of that you will be taken to a screen that approximates the Tools->Options IntelliTrace page in Visual Studio. Look for a blog posting in the future giving more detailed info about those various options, but for now we will just leave the settings at default values.
Now that we’ve turned on IntelliTrace settings for a specific set of test settings we need to start one of our tests using those test settings. To make sure you get the correct test settings you can right-click the test and choose “Run with options” from the drop down setting, then make sure that you have the correct test settings selected in the subsequent dialog. Note that IntelliTrace is available only for automated (coded and UI) and manual test types.
Now during your test run whenever you file a bug the last section of debugging data collected via IntelliTrace will be automatically sectioned off and attached to the bug. Note that you can still keep running your test and filing more bugs without losing any IntelliTrace information, as we are just saving off a copy of the most recent section of IntelliTrace data when a bug is filed. In the picture of the new bug window you can see the .iTrace file in the lower left in the Details tab.
Opening IntelliTrace data with Microsoft Test Manager
Now that the bug has been created and the IntelliTrace data has been associated with it all we have to do is open it up from Team Explorer in the Visual Studio IDE. Just open the bug (it will look the same as the picture above) and click the link to the iTrace file to open it up. There is more in-depth info on iTrace files here, the only real difference with this file is that the Test Steps section on the summary page is populated. This section will list all the test steps that were seen during this IntelliTrace section grouped by test case and test session. In the example I have shown below I didn’t mark any test steps either failed or passed so nothing is currently listed. If you double click on a test step debugging will start up at the location closest in execution to where that test step was marked either pass or fail. From there you can browse around all the IntelliTrace info collected to see if you can get started on a solution for the issue logged by the tester.