New book: How We Test Software at Microsoft

So much work goes into a book that it’s a treat to tell everyone about the result of our labor. However, it’s even better when an author shares his excitement, which is what Alan Page, a Test Architect at Microsoft, will be doing today.

Alan, Ken Johnston, and Bj Rollison recently published How We Test Software at Microsoft(448 pages, ISBN: 9780735624252) in Microsoft Press's Best Practices series for developers. The authors have also created a website devoted to the book: HWTSAM (Information and discussion on the MS Press release "How We Test Software at Microsoft”), where you can read reviews, review the book’s table of contents, and see pictures of the book (in the wild indeed!). And Alan’s guest-post contains a lengthy excerpt from the book. Enjoy!

* * *

Alan here. Did you know that Microsoft has nearly 10,000 software testers? Do you wonder what they all do? How do testers fit in with the other disciplines at Microsoft? What tools do they use?

People ask me those questions and many more like those all of the time. So, a few years ago when my manager (at the time), Ken Johnston asked me to write a book about testing at Microsoft I was intrigued. OK – to be completely honest, at first I told him to go away and leave me alone. There are hundreds of existing books about testing. I’ve read dozens myself, and I just didn’t see the use of adding another book to the testing pile. Then Ken reminded me about how we make the technical training we provide at Microsoft come alive. Sure, we talk about boring sounding things like Equivalence Classes, Boundary Values and Model-Based testing, but we tell stories about how the techniques and approaches are used at Microsoft. Books like Code Complete , and Writing Solid Code are great books because they are full of stories. Joel Spolsky has released two popular books full of stories about programming. So, the premise of the book would be that we would talk about how we test at Microsoft, but that we’d pack the book full of stories about where our techniques, tools, and approaches worked…and where they failed.

We broke the book into four sections. The first section includes some background information on how engineering teams are put together, what roles testers play, and how they all get together and go about making software. A lot of what we do in test at Microsoft is based on the context of how many products we make, how many engineers we have, and the roles the engineers play. The second section discusses some of the approaches we use for test design, including functional and structural testing approaches, code complexity, and model-based testing. The third section of the book covers some of the tools we use for testing, and for tracking bugs and test cases. Finally, we close the book with chapters on testing software services, emerging approaches at Microsoft such as leveraging virtualization for automation labs, and close with a chapter looking to the future of testing and quality at Microsoft.

One of the reviews of our book states that Chapter 8, “Model-Based Testing,” is too short, that the full story remains out there. That chapter always had the potential to explode, but we kept it to only 24 pages. If there was one chapter in the book that could turn into another book entirely, it would be that chapter. The model-based testing chapter is my favorite of the entire book, probably to some extent due to the inclusion of a story about my very first car:

Designing a Model

Creating models isn’t terribly difficult. It is often more difficult to know when to stop creating models. My first car was a 1962 AMC Rambler. It was fun to drive, yet quite simple. It had push button gears—one each for Drive, Low 1, Low 2, and Reverse. The Neutral button doubled as the starter. When I got in the car to go somewhere, it was always in the same state—parked and off. After I got in the car, I would take an action (pressing the green start button) to take the car to a new state—running! From here, I usually put the car into drive (yet another state) and went to school. Sometimes I forgot something and had to return the car to the previous state (off) while I ran inside to grab my homework, books, or whatever else I missed the first time I ran out the door. I could also go from gear to gear whenever necessary. Technically, the car could be put into reverse at any time (I never tested that), but for simplicity, let’s pretend that I had to put the car into neutral first, as shown in Figure 8-4.

This model (like my Rambler) is rather simple. The “problem” is that there are many more actions I could take with my car. The model in Figure 8-4 doesn’t take the actions of the brake or gas pedal into account and those could be modeled as well. The windows could be in varying states of openness, and the headlights could be in any of three different states (off, on, bright). The heating system, the windshield wipers, and the radio all had entirely different models, some, unbelievably, that interacted with each other. Models can grow quite quickly. As mine grow, I always recall a bit of advice I heard years ago: “ ‘Too small’ is just about the right size for a good model.” By starting with small models, you can fully comprehend a subarea of a system before figuring out how different systems interact. When I first learned to drive, I learned the basics of each part of the car (how to make it go, how to turn, and how to stop). As I gained experience and learned more about each subsystem of the car, I got better at driving. I never drew models of my Rambler, but I’m certain that in some sense, I thought of the different pieces of the system as models.

This excerpt reminds me of another theme that runs through the book. If you didn’t keep on reading beyond the excerpt above, you may not know what this had to do with testing. We do a lot of that in the book—we talk a lot about testing without talking about testing. We have stories of cooking, apples, fishing, and secret hideouts, and we still find a way to link them all to testing at one of the most successful software companies in the world.

To give you more of a sense of the book’s rhythm, here’s a bit of what follows my discussion of the Rambler in Chapter 8:

Modeling Software
A great deal of software is state-based and can benefit from a state-based testing approach. Figure 8-5 shows a simple application with three buttons. The first button prints “Hello” to a text box, the second prints “World” to a text box, and the third button clears the fields. When the application starts, both text boxes are clear regardless of their state when the application terminated.

As far as applications go, this one is as uncomplicated and minimal as you will ever find. The first thing a tester on this application might do is generate simple, serial test cases, as shown in Figure 8-6.

At first glance, you might think that there are only three actions that need to be tested, one for each button. Experienced testers might wonder, “What happens if I click ‘Say Hello’ twice in a row—or 50 times in a row?” (Good testers seem to understand that no matter how absurd clicking a button 50 times in a row might sound, somewhere, a customer will do exactly that.) Scripted tests for this application could look like the four tests listed in Table 8-1.

For this application, these test cases may be enough for adequate testing. There are, however, drawbacks to tests like this. First, these test cases require manual maintenance. If a tester forgets to add a new test case (for example, the tester later realizes that he needs to create a test where he closed the application without clicking any button), the tester would need to make updates to the scripts. More important, because these tests are static, they will always test the same thing. Although this is often a good (and expected) attribute (any regressions in functionality will be found quickly), many scripted test cases never find bugs
after they are created and executed once. In a sense, the tests train the software to succeed against the test cases. Figure 8-7 shows a state model for this application, along with bold lines to show which transitions and states in the model the tests in Table 8-1 “visit.”

A state model can clarify understanding of how the application works and what the different test permutations might be. Figure 8-8 contains a state model for this application. The four boxes represent the four possible states of the application, and the lines represent the actions that can be taken from each state. The states are as follows:

1. Both text boxes are clear (S1).
2. The first text box contains “Hello” and the second text box is clear (S2).
3. The first text box is clear and the second text box contains “World” (S3).
4. The first text box contains “Hello”, and the second contains “World” (S4).

 

The connecting lines represent actions that cause transitions from state to state. For example, from the start state (both text boxes clear), there are three possible actions. Clicking the clear button returns the application to the same state (or does nothing depending on your point
of view). Clicking either “Say Hello” or “Say World” changes the application to a new state.

A random walk through the transitions and expected states might look like what is shown in Table 8-2.

The chapter then continues with sections on building finite state models, modeling without testing, and model-based testing tools at Microsoft, as well as some inside views (that is, more stories!) by Microsoft SDETs.

I hope you’ll find the balance of stories and technical details in our book both enjoyable and helpful. Buy HWTSAM Now!

—Alan Page

* * *

You might also enjoy this recent interview with Alan. And here’s a nod to the book’s coverage of Equivalence Class Partitioning and Boundary Value Analysis. And, finally, Alan, you need to point us to what I assume is your acting debut.

More new books will be coming to the blog soon. I can hardly keep up. :-)

Share this post :