The Microsoft Test Runner – innovation for the Generalist Tester

It’s a very busy phase for us at work. The entire team is busy stabilizing our products for the upcoming VS 2010 Beta release. This is exciting – the team is really proud of the new Test and Lab Management products that we have been designing and building for months, and can’t wait to get the Beta bits into your hands. I spent a significant amount of my time last week taking our bits for a test drive, and have been waiting to find the time to write to you more about them. It’s Saturday and I am just back from a round of golf. The weather here in Hyderabad, India is extremely hot already (95 degrees – and we are still in February) I am ensconced in my study now, and have the next post ready in the series of “re-thinking software testing”. I hope you enjoy learning about our new tools, and look forward to test driving them soon yourself!

In the last post, I talked about Camano, our new test case management solution. Once the test plans are in place, test execution is the next focus area for testers. In today’s post I will introduce MTR – the Microsoft Test Runner.

In an earlier post I had mentioned that our research shows that 70% of the testing done in the industry is manual testing done generalist testers. As we talk to the generalist testers though, we find that they today are frustrated by two aspects of their job:

1. They spend a lot of time in non-core tasks e.g. filing a detailed bug, reproducing the bug, reporting test status etc.

2. There is a lot of repetitiveness in manual testing e.g. they waste a lot of time navigating to a particular point in the application they are testing, and repeating same test with different test data and configurations.

MTR (Microsoft Test Runner) was designed to solve these types of problems. Below is a screen shot of MTR, and I then talk to about its key features.


1. Help manual testers by showing detailed steps & logging results: When one or more tests are launched from Camano’s testing activity center, they are loaded in the Test runner. You can look at the test case with its steps. The detailed steps provide information about what you are supposed to do next, any expected results and attachments etc. You can mark the results of the steps as passed/failed. You can also add any comments. When you are done with the testing, you can publish the results to the Test Case Management (TCM) server. Reports can be generated from the TCM server for any level of information about the tests. You don’t need to separately generate reports of what you have done.

2. Create actionable bugs: The problem of no-repro is a big issue between developers & testers. Testers spend enormous amount of time finding and reporting a bug, but many of the bugs are rejected by developers because the information in the bug report is not comprehensive enough, or they can’t reproduce the bugs on their systems. MTR goes a long way in solving this problem. While using Test runner, if you see an issue, you can, with one click of a button, get the screenshot of the area on the application and attach it to the step. To file a bug, simply click on the create-a-bug icon it populates the bug form. The bug form automatically includes all the information needed to reproduce the bug including what steps have been tested; what exact UI actions were performed on the application (yes each and every UI action), the video of the testing being done; detailed system information about the machine used to test; the comments & screenshots added during test etc. (Of course you control what to record in test settings). You may just type the title of the bug and fire away! The screen below shows how a part of the bug form looks like.


3. Fast forward for manual testing: When the developer fixes the bug , you will need to verify the fix. You may need to go through a lot of steps to reach the point where you want to pay more attention. With the fast forward feature, you can ask Test Runner to navigate the application to the right spot quickly. How does it do it? While testing the first time, the action recording engine in Test Runner captures your UI actions and can use these actions to fast forward quickly to the right spot. You can skip from any point in your test and then test manually and then skip again & so on. The two screen shots below show the tester being prompted to select the steps to fast forward, and the system playing back those actions to get the tester to specific test she wants to focus on.



4. Data driving tests: As mentioned earlier, it is a laborious process to test the same test case with different data sets. But it needs to be done to look for bugs that may come with different data (for example, try wrong data, no data …). Fast forward functionality of Test Runner not only works with the actions you performed, but is also intelligent enough to differentiate between normal UI actions and data entry. When you test the same test case with different data set, fast forward functionality will pick up the right data sets to be entered in the app while navigating. This helps you focus on the app and find bugs rather than typing all that data.


5. Shared steps: Many test cases share same steps to test various functionalities. Test Runner allows you to share these steps in sharable units. That is, you can create shared steps and then use then in other test cases. Writing manual tests just become easier. If application changes and you need to update the steps, just update it in the shared steps and all test cases benefit from the change. Similarly, if you had recorded the steps (for fast forward functionality), the recording is also shared among test cases. So if the application changes (which does so often) its impact can be contained to just one shared component.


Can you believe we are still talking about manual testing?

I hope you like what you have seen so far – I’d love to hear from you, so please do leave a comment with your opinions, comments, and questions. 

In a subsequent post I look under the covers of the MTR with you and talk about the innovative way it works. I will also discuss about its extensible architecture. But before I do that, I will introduce our support for automation in the next post. Talk to you soon!

Comments (6)
  1. bob53050 says:


    As much as you are anxious to deliver the new beta bits to us, I am very anxious to see the progress on these new product areas which have a great deal of potential 🙂

    It is pretty cold yet in Wisconsin so we could try to ship some of our "coolness" to you 🙂

    • Bob Hanson
  2. chandoo says:

    Amit… its definitely not 95 degrees 🙂 it must be 95 Fahrenheit, good post by the way!

  3. I am very excited proud and proud to introduce the new testing features you’ll find in dev10. In addition

  4. With todays busy life schedules is just hard to take exams. With not enough time to even stand and stare, where do we get time to study and appear for the exams. Do you have all the skills  but lack the certifications to prove it??Just appearing in the exams without much preparation will be a complete waste of time and money. In the case of the competitive IT exams such as the MCSE, MCTS etc, passing the exam will be a tough task even if we do a little preparation. This is where the need for buying these IT certificates comes into play. “People out there buy IT certifications such as the MCSE because of many reasons. It is a well known fact that not all the people out there are brilliant enough to take the highly skill requiring exams such as the MCSE. MCSE is one of the most popular and premiere Microsoft certification standard, after securing which an individual is recognized to be able to analyze the business requirements for information system solutions, as well as design and implement the necessary infrastructure as required. Though the skills can be developed, getting certified in MCSE is difficult. This is one of the reasons why people search out  certification providers to fit their means.

    In regards to our 100% refund we are so confident you will pass the exam that we are willing to provide this option to you.If for some strange reason we are not able to execute your exams we will refund all your money minus the service that any institution charges for example(PAYPAL).This past year 2009 we manually tested more than 300 candidates all with 100% passing test scores.that means nobody failed.This year we intend to break our last year record and aim for a higher turnaround rate. Yes,there are other companies out there that provide the same service but we believe that  our service and professionalism surpasses all other companies out there .

    If having the real deal is not your cup of tea,we also provide fake certifications printed on 110 lb parchment paper.This is of course not the real Mccoy but you can use it to impress your friends some people will be fooled by its appearance.

    Thank you for taking the time to read this  is partner with several authorized testing centers all over the world. No matter which place you are located in, they can help you to take Exam in VUE/Prometric. You are able to receive your certifications at home without spending your time and effort!

  5. Anonymous says:

    Poor Article: Where are the references? For example, what is the link to Microsoft Test Runner?

Comments are closed.

Skip to main content