I think my team – much of Microsoft, in fact – is going about testing all wrong.
My team has a mandate from on high to automate one hundred percent of our testing. We aren’t the only ones either, I don’t think. This is a great goal (even if it isn’t immediately achievable), but that’s not what we’re doing.
We are taking manual test cases and scripting them. That’s it. End of sentence. Period. Full stop.
Now don’t get me wrong – we are taking “scripting manual test cases” far beyond what it normally means. Thanks to our Logical Functional Model our test cases are written from our customer’s point of view. Thanks to our Physical Object Model and Application Internals Model the details of interacting with our UI and yanking data our of our application are stuffed into small boxes rather than pervading our test cases. Thanks to Comprehensive Verification the intricacies of determining whether what happened is what really should have happened are stuffed into another set of small boxes. And thanks to Execution Behaviors and Data Providers we can cover the infinite variety of ways to do things by writing one test case and running it a bunch of times rather than writing a plethora of test cases that each vary from the other only in some small detail. So we can script many many more test cases than we would possibly be able to otherwise. But we are still just scripting manual test cases.
It used to be that it took a very experienced person to write scripted test cases because they had to deal with all these issues all at once. Decoupling the parts from each other makes building each of them simple. Scripting the test case itself is now something the average ten-year-old could do. But my team full of experienced testers still spends most of our time scripting new test cases and – the big time suck – keeping all those test cases we wrote yesterday and before running. (Even though fixing a, say, UI manipulation problem, only requires a few small changes, it still takes time to figure out the problem and how to solve it. And every other failing test case still has to be looked at just in case it is failing for some other reason.) Some of this maintenance pain is because we are working with a brand new application that is undergoing frequent significant changes and also building on top of a brand new platform (Windows Presentation Foundation), so there’s some hope that whenever we enter final stabilization things will settle down, and that they will stay mostly settled down as we build future versions. (I can’t even imagine how horrible it would be if we didn’t have our stack protecting us from most of this. (Actually I don’t have to – I’ve lived in that particular war zone. It’s not fun, nor is it pretty.)) But I am skeptical that all of these problems will just go away.
So why are we still spending our expertise on simple work? Let’s bring in contractors, or the less experienced testers management seems to be trying to get rid of, and let them do this. (Contractors because that’s what you pay a contractor for, right? To do work you don’t want to do? And less experienced testers because they could add huge value now plus have a clear growth path as they gain experience.) I want my experienced testers spending their time actually testing. I want my experienced testers figuring out how to automate their testing so they can move on to looking at the next level of problems. And then figuring out how to automate that testing and moving on to the next level. And so on and so on and so on.
I want my testers to test, not automate.
[Please remember this is all my opinion and does not necessarily reflect that of anyone else at Microsoft or Microsoft itself. And see the disclaimer in my sidebar.]
*** Want a fun job on a great team? I need a tester! Interested? Let’s talk: michhu at microsoft dot com. Great coding skills required.