As a result of writing the post: Edit Test Case properties directly from the test runner of Microsoft Test Manager I asked one the ALM MVPs, Terje Sandstrom, to also look at one of the most requested Visual Studio Update 1 features – but an area I didn’t have a post on yet: Grouping in the new Unit Test Explorer. He takes a look and by morning sends me this post and indicates he wants to do three more in this series!
Part 2: Using traits in TFS Build
Part 3: Testing and building using different test frameworks
Part 4: Managing code coverage for unit tests
Part 1: Using Traits in the Unit Test Explorer by Terje Sandstrom
This is the first post in a series on Unit Testing in Visual Studio 2012, focusing on the improvements in Update 1.
Visual Studio 2012 has a great new Test Explorer. The Test Explorer is where you see the results of your unit tests. In Update 1 it has been extended, and you can now organize and filter the test runs based on several conditions, among them your Projects, and Traits. Traits is a new concept, a common denominator for several underlying terms, Test Category, Test Property, Priority, and Owner.
Traits are not only useful as a grouping mechanism in the Test Explorer, it also part of what can control which tests you run. This has been a big request – because it means the developer can focus on the tests relevant to the work, and not be bothered with running other, perhaps long-running tests, which would otherwise slow down the whole development experience. No longer with Update 1!
Traits are also used in TFS Build as one of the criteria to determine which tests to run for the different builds there. TFS Build is not “trait-aware” in itself, but TFS Build can act on the underlying types, Categories and Properties. The details are in Part 3 in this series of posts.
Using Traits and Projects for grouping
Let us start with a MSTest based example. In a sample project we add 3 tests, and put them into separate categories. This is done by adding the TestCategory attribute to the method, and the name of the category as a string.
Open up the Test Explorer, and choose the Group button. Compared to RTM we have now got 2 more options, Project and Trait.
Under the menu the three test methods are listed based on Outcome. Changing that to Trait gives us the test grouped by their MSTest Category.
The other option is to group by Project, and adding a new test project it looks like this:
Using more trait concepts
As mentioned above, Traits can also be based on some other properties. Owner and Priority are explicit and straight forward enough, whereas TestProperty gives you a property-value system. That means you can extend the Traits with your own concepts using TestProperties.
In the code below, we have added these properties, and also a new property called “Time”, to indicate if it is a fast or long running test.
In the Text Explorer, still grouping by Trait, this now looks like:
Notice that the same test method appears multiple times, but this is just visual, they are only run once, so the total time here is not 57mS but still 19mS.
Did I say “just visual” ? It is in fact more than that .
Using Traits to control test running
The Search field is named and looks like a search field. But a better term for it is that it is a Filter controlling what to run. If you click the small arrow to the right, a drop down menu appears.
You see a set of possible Search Filters, and above that a recent list which contains a previous set filter.
Filtering on trait “CI” gives us this list:
The filter term will match any trait you have, be it in a category, a priority, a owner or in a property. In fact it matches also these words, so you can set a filter to
and it will match all tests that have any priority set.
The filter set will stay there until you change or clear it, or restart Visual Studio.
With this filter set, the Run All button now runs ONLY the tests within this filter.
This also applies to the automatic test runner – I assume you are aware of that one ?
The automatic test runner
The icon to the upper left is enabled here, it means that the background automatic test runner will run every time a build is performed.
Make it a habit of pressing Shift-Ctrl-B instead of just saving your work. This keyboard shortcut will do a save all and then build your application.
Enabling the automatic test runner can be done either from this button in the Test Explorer or from the menu:
Control test runs based on other filter terms
The filter menu allows you to set up filters for not only Traits and Projects, but also for other aspects. The table below summarizes these with the syntax to use:
|Error Message||Message: <any part of text after message output>||Message:”Assert.Fail”|
|Output||Output: <any part of the output text>||Output:”Result”||Assume a trace command sending out “Result is”+resultvalue|
|Fully Qualified Name||Fullname: <namespace.classname.methodname>||FullName:”UnitTestProject1”||Takes all tests in that namespace|
|FilePath||FilePath: <any part of the file path name>||FilePath:”Integration.Test”||All tests with this in the name. Includes folders above the file too, that is the file’s fully qualified name.|
A filter expression can be negated by placing a ‘-‘ in front.
Filter expressions can be combined, that is, you can add more together, just separate them by spaces. The combined filter expression is the logical AND of the two expressions.
If we look at first only tests for CI, we find two tests:. In
Adding another search term, looking for only tests with Owner = Terje, reduces the list.
The most typical use for the logical AND is to handle property value settings. If you want to explicitly filter on Priority = 2, and be sure not to filter on anything else which uses the number 2, that could be expressed as:
By combining the different traits you should be able to limit the number of tests you run locally. Pay attention to creating a set of useful and unique trait names, categories and property/value based ones. This will make it easier to later set up filters. Also take into account that you use the same traits when filtering unit test cases for running in different builds. We will cover this in Part 3.
Also note that these traits are not limited to MSTest, but works just as fine with the other test frameworks, we cover that in Part 2.
Some useful links:
- Peter Provost has written a great blog post on the features of the new Unit testing in Visual Studio 2012.
- Running unit tests with the Test Explorer is described in this MSDN article.
- The different unit test frameworks that can be plugged in, is listed in this post.
- Installing the unit test frameworks is described here.
- Controlling code coverage is described here,
- Modifying the runsettings is here.
- Details on Test Categories
—— Chief Software Geek at Inmeta Consulting in Scandinavia —– and a Visual Studio ALM MVP