Visual Studio Team System (VSTS) 2010: Making Testing Easier
During the Mark Mydland tour, we got a glimpse of the new features in VSTS 2010, with an emphasis on the new features around testing for both developers and testers. Here's a quick summary of the cool features coming specifically in the dev and test editions.
First, a little background. At Microsoft, we have 2 different breeds of testers: Software Test Engineers (STE) and Software Design/Development Engineers in Test (SDET). In testing terminology, STEs are generalist testers, and SDETs are specialist testers.
Generalist testers are testers in the organization who have deep domain knowledge in the rules and processes of the business as well as expertise in the art of software testing. Generalist testers do not generally possess deep technical knowledge and therefore accomplish most of their testing via interaction with the UI of the application (also called "manual testing").
Specialist testers are testers who are highly technical and regularly write code. They are usually responsible for automation and white-box testing.
Some fun facts from the Visual Studio team from their research on how people test:
- 70% of all testers are manual/generalist testers.
- 90% of all testing is done manually.
Many of the features of the 2010 release are aimed at improving the experience for manual testing, or testing via interaction with the UI of the application, as generalist testers do.
Test Case Management Client (Codename: Camano)
Camano adds a new tool to your testing arsenal for creation, management and execution of tests. While specialist testers will continue to use Visual Studio to create specialized tests like Web, Load, and API automation tests, I expect that generalist testers will use Camano as their one-stop shop for testing. Camano connects to the new Test Case Management features of TFS to store and manage test artifacts including Test Cases, Test Suites, Test Plans, and so on.
During exploratory testing, Camano silently collects data while allowing the tester to interact with the UI of his application. When the tester identifies a bug, Camano transfers the collected data to the bug in the form of detailed repro steps recorded from the tester's own actions (not just the steps that were in the test case, although those are captured too), video of the tester's desktop during test execution, system information from the tester's machines, and a historical debug log with details of the execution (discussed in more detail below). So Camano does most of the work of creating a bug report for the tester, and the developer gets a wealth of information on how to repro the bug! You don't need instrumented builds to do this.
You can use Camano to test both browser and client (WinForm, WPF, etc.) applications.
Note that Camano is a separate application from Visual Studio, not an add-in. Therefore, a generalist tester could buy just Camano, for a lower price point, and not need to buy Visual Studio.
Automation for Navigation
Camano also provides low-overhead UI automation. Camano records the steps that the user took while executing a test the first time through. Should the test be rerun to verify the regression or to validate the fix, the tester can choose to have Camano rerun the test. This feature expects that the user is present during playback as it allows for no explicit validation.
Automation for Navigation is based on the Microsoft-internal RPF (Record and Playback Framework?) technology that has been around for 8 years. It's using Microsoft Active Accessibility (MSAA) and validation markers under the covers. The technology is currently used by Developer Division, Windows, and several other divisions at Microsoft (SQL Server, Exchange). The tool allows multiple technologies to plug in to provide technology-specific information. In VSTS 2010, the technologies will include WinForms, WPF, IE browsers, Firefox as a beta (at RTM time). Silverlight, additional browser technology, and partner controls will be in a future release.
Historical debugger (Codename: Proteus)
The historical debugger uses the CLR Profiling API to collect a debug log which can be opened by the developer in Visual Studio Team System in order to replay the test execution through the debugger as if the tester were running the tests live. It uses the PDB file, so the developer sees the code as it was when it was built, even if she has since modified it.
So here's the scenario: a tester finds a bug and sends a historical debug log to the developer (if the tester is using Camano, this can be created automatically by the tool). The developer can now replay the test and debug the code, stepping through it line-by-line in the debugger. How cool is that? As a developer, there were times when strange behavior would occur sporadically on my tester's machine, and I would give anything to be able to step into it and see what was going on, after the error occurred.
Test impact analysis (Codename: Aftershock)
Test impact analysis uses the CLR Profiling API to create a mapping of test cases to methods executed. Also, it uses a combination of source changes and build outputs to build a mapping of methods changed in a given build and by extension across builds. With this information, Aftershock identifies test cases that are suspect (i.e. the code executed in running the test has changed) for a given build being moved to test. So when a developer makes a code change, you can see how many tests that really affects (we all know - seemingly minor changes can have unexpected side effects). For many test organizations, this will mean a deterministic way to identify the minimum subset of tests which should be re-executed prior to final sign off.
At Microsoft, many products have a large number of automated test cases (between the millions of different configurations of different OS/browsers/languages/etc.). Aftershock is being used by Microsoft to limit the number of regression tests that we have to run.
Cool related work from Microsoft Research
I'll do separate blog posts on Pex and Chess. These will not necessarily ship with VSTS 2010, but are currently available through DevLabs.
Can I try this stuff out?
Sure. You can download the CTP of VSTS 2010 from https://www.microsoft.com/downloads/details.aspx?FamilyId=922B4655-93D0-4476-BDA4-94CF5F8D4814&displaylang=en.