Test Configurations in a Test Plan

For many, many test teams, we hear that they don’t typically test on more than one configuration. This is a big departure from what we at Microsoft are used to. Obviously we have many operating system versions including languages, versions of IE, versions of Office, etc. We’re used to dealing with testing on many platforms because of the configuration matrix.

So for testing teams that don’t use different configurations, we are here to make your life as simple as possible. There’s no reason to make your life harder than it has to be. For you, we have an out of the box, default configuration. You can edit it if you want, or just ignore it entirely. Go on about your testing, and don’t worry about configs.

For all the poor souls who do need to test on multiple configurations, we haven’t forgotten you either. We use MTLM ourselves, so it must meet our own needs too.

Creating configurations is fairly straight forward. First, you visit the Test Configuration Manager (in Organize), and open the Edit Test Variables dialog. In here you can define the different factors you will have variations in, such as OS, Browser, language, etc. Then for each factor, define the variations in the right pane. Then you go back to the manager and begin defining test configurations. Give it a descriptive title, and then add the relevant factors to the grid at the bottom, giving each the correct value. This will be helpful later in reporting where you can look for test result trends across the factors rather than simply reporting on each independent configuration.

The other important value on a test config is whether it is the default for new test plans. If it is set to default, whenever a new test plan is created it will automatically include those configs. Yes, you can remove project defaults from your plan, and add non-defaults. The goal for the person defining the test config project defaults is to make sure new plans have the configs you expect.

Test config defaults on the plan are automatically applied to test cases added to the plan. In Test Plan Properties, you’ll find a test configuration selector where this is specified. Again this is meant to make your life easier, so plan ahead and make sure you have the configs set up on your plan before test cases are added. If you get it wrong, don’t worry, you can change configurations applied to test cases after the fact.

There’s yet another level of ways to scope defaults. We’ve already talked about project-level defaults, test plan defaults, and now we’ll introduce test suite level defaults. Any new suite in the plan will inherit from the parent. That can be changed so that all test cases add to it get a different set of configs applied. In the suite header in Test Plan Contents, look for a drop down next to the label that says ‘Test configurations (x):’.

Finally, test cases can be changed independently of the plan and suite. Just select the test cases in question, and click the Configurations button in Test Plan Contents.

One last thing, when you are looking at test cases in the suites in the Test Plan Contents activity, you’ll see one instance of the test case. However, since many configurations can be applied to that once instance of the test case in the suite, these entries will be multiplied when they are viewed in the Run Tests activity. This is because the test case needs to be run multiple times, and selected independently from the other instances so they can be run with other similar configs. To assist in that work flow, you can use the Filter button on the Run Tests suite tree view toolbar to only show a specific config. Now you can focus on testing just on that one config, then change the filter later when you are ready to move onto another config.

Cheers,

David Williamson

Engineering Lead, MTLM