Do SDETs dream of electric sheep?

I read Joel Spolsky's blog from time to time. I am a little embarrassed to admit that I took the following entry a bit personally.

Apparently—and this is all based on blog rumors and innuendo—Microsoft has had a long term policy of eliminating all software testers who don’t know how to write code, replacing them with what they call SDETs, Software Development Engineers in Test, programmers who write automated testing scripts.

The old testers at Microsoft checked lots of things: they checked if fonts were consistent and legible... they checked whether the screen flickered when you did things, they looked at how the UI flowed, they considered how easy the software was to use, how consistent the wording was, they worried about performance, they checked the spelling and grammar of all the error messages, and they spent a lot of time making sure that the user interface was consistent from one part of the product to another, because a consistent user interface is easier to use than an inconsistent one.

None of those things could be checked by automated scripts. And so one result of the new emphasis on automated testing was that the Vista release of Windows was extremely inconsistent and unpolished. Lots of obvious problems got through in the final product… none of which was a “bug” by the definition of the automated scripts, but every one of which contributed to the general feeling that Vista was a downgrade from XP.

- from Joel On Software

I wanted to take a little time to cut through the blog rumors and innuendo and shed some light on the SDET role at Microsoft.

I don't think it's really accurate to call the automation test that we write "scripts". We write programs to test programs, and sometimes those have quite a bit of complexity. We have programs that put Windows Mobile through its paces for hours at a time, making calls, browsing the web, and sending e-mail. Beyond obvious things like calling APIs with various arguments and checking their return values, we have automation that runs features with enormous data sets and measure its performance. We have automation that runs through entire end user scenarios from end to end and verifies that it all works. We have automation that can check each screen of the product against a database of past saved images and e-mail me if a single pixel moves out of place. In some situations, writing the automation to test a feature can be more technically challenging than writing the feature in the first place.

In the end, our software isn't used by robots. People buy our software and people use it, so we need to make sure that people will like it. Joel is totally correct that there are some dimensions that are difficult to measure with automation. We use a variety of means to mitigate that risk, like

  • bug bashes - a broad set of people set aside a block of time to explore the product or dig into a specific feature
  • ad-hoc testing - spending time exploring your feature manually or with the aid of tools
  • end user betas - wide scale beta testing for external users
  • enterprise TAP programs - focused betas for our enterprise customers
  • usability studies -  we bring in users and observe how they interact with the product
  • and dogfood - we use the product to do our jobs, every day

The dogfood program is probably one of our best sources of bugs like that - if you think the flames and vitriol directed towards our products on the Internet are bad, you should feel the wrath of a Microsoft employee who is cranked that something is too slow or takes too many clicks to use. Factors that Joel mentions like perf, usability, consistency, and flow are all in bounds for the developer, program manager, and tester to file as bugs against a feature, but we also get plenty of feedback along those lines from the dogfood programs. They're definitely all "bugs" by our definition of quality.

As an SDET you get to free reign to write code that will push our products to the breaking point as well as putting yourself in the customer's shoes and making sure that we delight them. Does that interest you? We'd like to hear from you.

Further reading:

Nitpicker's Corner:

  • I didn't work on Vista. I can't make authoritative statements about it.
  • I didn't make the decision to migrate from STE to SDE/T and I can't make official comments on it.
  • It's very likely that I can't do anything about a particular dialog box that may be misaligned in a Microsoft product.

Comments are moderated - please be civil and on-topic.

Scott - SDE/T, Windows Mobile Security