DEV301 - Visual Studio 2005 Team System: Enabling Better Software through Better Testing
Presented by Tom Arnold
[UPDATED] - Visual Studio 2005 Team System: Enabling Better Software through Better Testing white paper AND Team Tester Edition Demo Video on MSDN
[UPDATED] - Brian Malcolm's Blog | Chris Lucas's Blog | Dominic Hopton's Blog
I missed the first few minutes of the presentation, but I suspect it was Tom introducing himself and saying a few words about Team System.
Tom was talking about the different types of tests you can author in Team System.
There are four test types that ship with Team System: Unit, Web, Load and Manual.
There's actually a fifth "type", but it's basically a mechanism for executing third-party test tools that aren't integrated in Team System.
Tom provided the background of the sample application he planned to use for the talk.
It was the same scenario as the DEV200 general session keynote on Tuesday - adding new functionality to an existing fictitious site called Adventure Works. The application consists of both a Web application and a Windows application.
Fast forward to the time in the life cycle when code is being written and people want to start testing stuff.
Previously, you had to go outside of Visual Studio to create any kind of test.
Now, Team System has a new project type - a Test Project.
Tom tempted the demo gods by using Visual Basic instead of his much-practiced C#.
Fortunately, the demo gods were not upset and it still worked.
From within a Test Project, you can add any of the test types mentioned earlier, except for unit tests.
Adding a unit test is done from within the project under test.
DEV301-01 - Unit Test Configuration Dialog
When creating a unit test, you can select which classes and methods to test in the unit test
Unit tests generate a TestClass attributed class.
There are several new attributes associated with unit testing.
DEV301-02 - Generated Unit Test Code
Attributes used by default in generated code:
TestClass classes - the class that contains the unit test
TestInitialize methods - code that will run before the test to establish pre-conditions
TestCleanup methods - code that will run after the test to perform clean-up operations
TestMethod methods - code that exercises the code under test
There is also an Assert class to enable the unit test to validate if certain conditions exist.
(e.g., Assert.AreEqual, Assert.AreNotEqual, Assert.Inconclusive, etc.)
Generated unit test will compile without any additional work, but you'll need to write some code in the TestMethod methods to get some real useful work out of them.
The Test View shows the tests you've created for a project. It's a lightweight user interface similar to the Properties Window.
When executing a test, you first see the Select Test Run Configuration dialog box. In this dialog you specify where the test should run among other things. Also, you can enable code coverage instrumentation from this dialog.
DEV301-03 - Run Configuration Dialog
The Test Results Window shows status of tests as they are executing (Pending, Passed, Failed, Inconclusive).
You can right-click a test in the test results window and file a bug and attach the test results from that test.
Code Coverage results are also available in the Test Results Window as percentages of lines covered in each class and method. You can also see highlighting in the source code edit window showing which code was actually covered and not.
Creating a Web test is relatively easy.
When you select to add a Web test to a test project, Internet Explorer opens to record the Web test.
The Web Test Recorder bar on the left side of the browser tracks the URL's visited.
DEV301-04 - Web Test Recorder Bar
When you close the browser, the Web Test Player appears in Visual Studio.
The Web Test Player allows you to step through the Web test.
DEV301-05 - Web Test Playback in IDE
As you step through the Web test, you can also review the request and response headers and HTML
To make your Web test more dynamic, you can easily add a data source to data bind values from a database to the
Web test parameters.
DEV301-06 - Databinding - Selecting a Datasource
DEV301-07 - Databinding - Selecting Data Fields
DEV301-08 - Databinding - Binding Form Parameters to Data Fields
At this point, Tom provided an aside story of how he and another guy did load testing on a major e-commerce site prior to the holiday season. At this particular company, they had no choice but to use the production servers to conduct their load test. Since they were using a production system, the shipping department was warned not to process orders that arrived with a special coding. Of course, the orders from the load testing arrived without said coding and they shipped 700 Easy Bake Ovens to an address in Walla Walla, Washington. Kinda funny.
Anyway, Web tests let you replace form post parameters by databinding those fields to a datasource.
A Load test is just a Web or Unit test run multiple times using a particular profile.
Simply put, a Load test is merely a container of tests.
DEV301-09 - Load Test Configuration
When configuring a Load test, you can customize a number of profiles to simulate real-world behavior.
You can create user profiles to simulate think time, cache control, session cookies, persistent cookies, and IP switching.
You can create browser profiles to simulate users accessing the sites from a variety of browser types.
You can create connectivity profiles to simulate a percentage mix of connectivity speed.
DEV301-10 - Connectivity Profile
You can add multiple machines to run the load test to place heavy loads on your Web applications.
You an select which performance counters to capture and monitor during a Load test.
DEV301-11 - Performance Counters
You can create a load profile (constant of stepped #users, duration, step user count).
You can also do a lot of test configuration such as how long to run the test.
You can set threshold rule settings on performance counters so that you can flag critical and warning thresholds
Default critical and warning threshold settings are based on the load testing team's own experience testing real-world applications.
You can monitor performance counters in real time and drag them to the graph for instant graphing (like Performance Explorer).
Most importantly, you can bring all of that data to one desktop to monitor and control the load test.
DEV301-12 - Test Results View
DEV301-13 - Load Test in Progress
DEV301-14 - Performance Counter Exceeding Threshold
DEV301-15 - Code Coverage Results
DEV301-16 - Code Coverage Highlighting in Source
Manual tests in Team System are persisted as text or in Microsoft Word documents (.mht's), which enable you to fully to embed graphics and test steps.
DEV301-17 - Manual Test Authoring
DEV301-18 - Manual Test Execution
Team System provide a template with description, test target, version and test steps.
Manual test show up in the IDE when running a manual test and you can mark the test as passed or failed.
Integrated Third-Party Testing Tools
Compuware TestPartner tool integration was demonstrated by Mark Eshelby.
Test Partner is a functional test automation tool; capture/playback with VBA for easy scripting; works with Web forms and Windows Forms; also, it supports testing of Web, J2EE, ERP/CRM and distributed client/server applications.
DEV301-19 - Compuware TestPartner Integration
DEV301-20 - Compuware TestPartner script
Mark related his own load testing war story where he was once involved in a load test where they used an actual e-mail address that resulted in 20k e-mails going to one customer.
Test Results window shows results from third-party test tools that are integrated in Team System.
The TestPartner log even displays within Visual Studio.
DEV301-21 - Compuware TestPartner Integration in Test Results View
DEV301-22 - Compuware TestPartner Results in IDE
Back to Tom.
Test View is not useful for managing lots of tests.
Team System also has a Test Explorer, which enables you to create test categories (e.g., BVT - build verification tests).
Using the Test Explorer you can view all tests, tests by category, or you can specify a custom filter.
You can run an entire category of tests with one mouse click.
DEV301-23 - Categorizing Tests
DEV301-24 - Test Explorer Showing Categories
The Run Configuration Dialog enables you to run tests distributed across a lab full of machines.
And, you can view details of any test under progress from one machine.
You can execute tests locally and remotely using the IDE, command line, or Microsoft Build Tasks.
As part of Team System, the test tools integrated into the bigger picture: source code control of tests, issue/work item tracking, MSF, etc.
Test Reports and Analysis
On the Project Site, you can view various reports, such as a Project Health Summary Report and you can drill-down into reports for detailed information.
Some of the standard reports include, Dev Stability, Test Effectiveness, Team Productivity, and Project Quality.
Reports enable you to discover things about your project.
For example, if code coverage is not keeping pace with adding code, you can see you're not keeping up with right test coverage.
DEV301-26 - Software Project Health Report (no image DEV301-25 - dupe of this one)
DEV301-27 - Report Drill-down
You can also view builds reports, which enable you to see the status of builds and the results of BVT's.
Build report details include: projects in build, test results, work items included in this build.
DEV301-28 - Build Status Report
DEV301-29 - Build Status Report Detail
Support for multiple roles: test author (dev or tester), test execution, build lab support.
User interface new windows:
- Test View (author)
- Test Explorer (execute)
- Test Results (outcome)
- Code Coverage (test effectiveness).