Microsoft Test Runner series – part 1 – History & Goals

Article
01/12/2010

Hi everyone,

I hope you are using the MTLM beta2 release & finding it useful. I wanted to blog a few posts & talk about the Microsoft Test Runner component of the product that is used for execution of manual tests. I will talk about how it got started & what functionalities it provides to make the life of a generalist tester better.

History: When I joined the team, our goal was to define the vision of the product. We set out to find the pain points of testers. As we went to various companies, we saw that a lot of testing is still manual. All the testers were very professional & the felt proud of finding bugs during product cycle, so that the real end users don’t have to deal with them. It was a very good experience to meet such dedicated testers.

When observed closely, we saw that a lot of their time is spent on mundane tasks e.g. adding details to a bug, verifying & re-verifying that what they found was a real bug; testing the same test case over & over again with different data sets etc. While performing the steps, many of them had the high level steps written/printed by their side. Sometimes it was in the form of a printout from a word doc or excel sheet & other times they had the soft copy open on their computer & were flipping back & forth between the steps/instructions & the Application Under Test (AUT). A lot of time was also spent on reporting what they had tested.

We decided that we will focus a lot of energy in making the life of generalist tester a bit better & productive. Our goals were to let the tester focus on finding real bugs & reduce the mundane tasks that they need to perform.

The first goal was to reduce the ping pong between developer & tester about what the bug is & is it really an issue. We wanted to create an actionable bug that provides enough information to developer for him to fix the bug & not ask for more details from tester. We automatically added information like repro steps, detailed action logs, video of the testing, system information & data collector logs etc. Together these would help the developer get to the root cause of the issue, without tester needing to spend a lot of time filling this information in the bug.

The other goal was to provide a way for tester to see the steps & perform the actions on AUT. However, it was decided that the solution should be such that the tester can focus his/her main attention on AUT & just glance at this app whenever needed (e.g. to see next step or expected results). The interaction with MTR should be minimized as every interaction with MTR is actually taking tester’s attention from AUT & from finding bugs. Various strategies were considered e.g. providing a full screen view in a browser OR provide a minimal view shown as only one row. Ultimately, we decided to go with the small vertical app that is shown on the side of the tester’s machine. This view of Microsoft Test Runner not only shows the steps information, but also let’s user do most common tasks, e.g. take screenshots, file bugs & so on. To reduce the need to move the mouse between AUT & MTR, it supports global shortcut keys for common operations such as move to next step, pass/fail a step etc.

To reduce the effort required by tester while reporting the status/results, MTR allows testers to mark the result of the step/test case, add comments etc. while performing the test. This information is uploaded to server, when users save the results. This reduces the need for the tester to spend time in filling detailed status reports etc. Lead/manager can view the results anytime & can generate various reports.

The next goal was to reduce the time it takes to test a test case. One easy way out generally is to automate the test case. However, in reality it is not that easy or straightforward. Firstly, the tester needs the knowledge & skills to create the automation. Secondly not all tests can be automated completely (not worth the ROI). Thirdly, we wanted to provide the flexibility of choosing what portions to automate vs test manually. Generally, this information can’t be planned in advance, i.e. as a tester I want the flexibility that today, I want to test one section manually & the other section I want to skip, as it is not that interested Or is pretty stable. This way I can focus my energy on what I think may have most bugs.

Accomplishing this goal was the tough one. We argued about various choices (some that are being followed in industry). After lot of thought, we decided to try something new. Use Record & playback technology for fast forwarding the sections that are not very interesting. R&P has been used earlier & has it’s own issues – i.e. may not be used in all scenarios of automation. However, in this case we were talking about fast forwarding some sections of the test case. Also, there was some advance techniques that were being used within the company & had proven to be much more stable than myths that surround R&P. As this topic is long, we will talk about more in a separate blog how the tool is using R&P to provide fast forwarding capability for testers.

The other goal was to provide more insight into what went in the builds & what is impacted by the changes made to the code. Based on previous experience, we knew that one of dilemma testers face is when a small change is made to the code before a major release. The question comes, what should be tested & what can be omitted. If major portion of testing is done manually, then it becomes more difficult question Developers always say it’s a small change & will not impact any functionality. As if the testers can believe that J. So we came up with a solution to provide test impact of the code changes that helps testers prioritize the testing.

Lastly, one of the question that every test team need to answer is “Are we ready to ship?”. The solution needed to help the team answer that question.

In next few blogs we will talk about the solutions that the tool provides in detail & how it tries to fulfill the goals we set out with.

Microsoft Test Runner series – part 1 – History & Goals

Additional resources