I Want Testers, Not Automators


I think my team – much of Microsoft, in fact – is going about testing all wrong.

My team has a mandate from on high to automate one hundred percent of our testing. We aren’t the only ones either, I don’t think. This is a great goal (even if it isn’t immediately achievable), but that’s not what we’re doing.

We are taking manual test cases and scripting them. That’s it. End of sentence. Period. Full stop.

Now don’t get me wrong – we are taking “scripting manual test cases” far beyond what it normally means. Thanks to our Logical Functional Model our test cases are written from our customer’s point of view. Thanks to our Physical Object Model and Application Internals Model the details of interacting with our UI and yanking data our of our application are stuffed into small boxes rather than pervading our test cases. Thanks to Comprehensive Verification the intricacies of determining whether what happened is what really should have happened are stuffed into another set of small boxes. And thanks to Execution Behaviors and Data Providers we can cover the infinite variety of ways to do things by writing one test case and running it a bunch of times rather than writing a plethora of test cases that each vary from the other only in some small detail. So we can script many many more test cases than we would possibly be able to otherwise. But we are still just scripting manual test cases.

[Note that I am not saying scripted test cases aren’t important. They are. See Adam’s Myth Busting articles for some of the reasons why.]

It used to be that it took a very experienced person to write scripted test cases because they had to deal with all these issues all at once. Decoupling the parts from each other makes building each of them simple. Scripting the test case itself is now something the average ten-year-old could do. But my team full of experienced testers still spends most of our time scripting new test cases and – the big time suck – keeping all those test cases we wrote yesterday and before running. (Even though fixing a, say, UI manipulation problem, only requires a few small changes, it still takes time to figure out the problem and how to solve it. And every other failing test case still has to be looked at just in case it is failing for some other reason.) Some of this maintenance pain is because we are working with a brand new application that is undergoing frequent significant changes and also building on top of a brand new platform (Windows Presentation Foundation), so there’s some hope that whenever we enter final stabilization things will settle down, and that they will stay mostly settled down as we build future versions. (I can’t even imagine how horrible it would be if we didn’t have our stack protecting us from most of this. (Actually I don’t have to – I’ve lived in that particular war zone. It’s not fun, nor is it pretty.)) But I am skeptical that all of these problems will just go away.

So why are we still spending our expertise on simple work? Let’s bring in contractors, or the less experienced testers management seems to be trying to get rid of, and let them do this. (Contractors because that’s what you pay a contractor for, right? To do work you don’t want to do? And less experienced testers because they could add huge value now plus have a clear growth path as they gain experience.) I want my experienced testers spending their time actually testing. I want my experienced testers figuring out how to automate their testing so they can move on to looking at the next level of problems. And then figuring out how to automate that testing and moving on to the next level. And so on and so on and so on.

I want my testers to test, not automate.

[Please remember this is all my opinion and does not necessarily reflect that of anyone else at Microsoft or Microsoft itself. And see the disclaimer in my sidebar.]


*** Want a fun job on a great team? I need a tester! Interested? Let’s talk: michhu at microsoft dot com. Great coding skills required.

Comments (6)

  1. I agree. I ranted on something related today:

    Unhealthy Goals

    http://www.kohl.ca/blog/archives/000114.html

  2. Mark Eichin says:

    So what do you really mean by <i>actually testing</i> ? Do you just mean the exploratory work to identify things that can be tested?

  3. micahel says:

    Exploration is part of it, but also:

    – Thinking about what the app should do in a certain situation.

    – Thinking about things that are likely to not work correctly.

    – Building a model of the app (in your head, on paper, in a model-based testing tool, however) that you then use to drive your testing.

    – Testing around bugs to find other bugs.

    – Reading source code to reverse engineer the model of the app the developer had, and then using that to drive your testing.

    – Dreaming up new test cases of whatever shape and form.

    Thinking up a scripted test case is testing. Writing it is automating.

  4. MSDNArchive says:

    I could not agree more. That’s why I like <A HREF="http://fit.c2.com">Fit</A&gt;. With it, a product owner writes an EXECUTABLE spec. Developers then use that document to TDD their solution — at a functional test level. (They still do unit testing, too.) This way, the tester’s role become less about automating spec compliance and more about actually testing the software: exploratory, perf, scale, and all the other things that don’t happen when testers are too busy automating the happy path.

  5. I’m not so excited by Fit[nesse]. It has its usefulness, to be sure, but I hear very little critical discussion about it.

    1) Tables are a reasonable way of modelling some behaviours, but they’re not universally good.

    2) Fitnesse sometimes presents an interface that filters or limits input in ways that I would like to be unfiltered. One of the things that I like to do is to throw colossal amounts of data at various functions. Fitnesse kaks for me at 60000 characters or so.

    3) In my organization (and I bet in others), it’s easy for the developers to fall into the trap of believing that fixtures are the only form of testability that they need to supply, and they can get resistant when asked for other work.

    4) Fit[nesse] tables can fall into exactly the same trap as Michael describes above. I spend a lot of time on the care and feeding of the tool and its format. I find it seductive; I can waste time on it without noticing that I’m doing Fitnesse stuff, and not testing.

    Now, don’t get me wrong: it’s an interesting tool, a big philosophical step forward, and I find it to be helpful in exercising certain kinds of mid-level functionality. But I would like to hear it discussed with a little more circumspection, tempered enthusiasm, and thoughtful counterexamples of contexts where it doesn’t… Fit.

    —Michael B.

  6. Patrick comments on my When To Automate post:

    I am confused by your third bullet in that unstable/newly…