measuring testers

Yeah, I know … scary subject. But as it is review time here at the empire, this is a subject that has been front and center for both testers and the managers they report to so I’ve been asked about it a lot. I always give the same advice to test managers, but I’ve done so with much trepidation. However, I suddenly feel better about my answer because I’m in good company.

Before I give it away, let me tell you why I am feeling better about my answer. I came across a quote today while looking at the slides that Jim Larus is using for his keynote tomorrow at ISSTA (the International Symposium on Software Testing and Analysis). The quote captures exactly my advice to managers here at Microsoft who ask me how to rate their SDETs. Moreover, the quote comes from Tony Hoare who is a professional hero of mine and a friend of my mentor Harlan Mills (and a Knight, a Turing Award winner and Kyoto Prize winner). If Tony had said the opposite, I would have a whole lot of apologizing to do to the many test managers I’ve given this advice to. Whenever we disagree, you see, I am always wrong.

So here’s my advice: don’t count bugs, their severity, test cases, lines of automation, number of regressed suites or anything concrete. It won’t give you the right answer except through coincidence or dumb luck. Throw away your bug finding leader boards (or at least don’t use them to assign bonuses) and don’t ask the other testers in the group to rate each other. They have skin in this game too.

Instead, measure how much better a tester has made the developers on your team. This is the true job of a tester, we don’t ensure better software we enable developers to build better software. It isn’t about finding bugs because the improvement caused is temporal. The true measure of a great tester is that they find bugs, analyze them thoroughly, report them skillfully and end up creating a development team that understands the gaps in their skill and knowledge. The end result will be developer improvement and that will reduce the number of bugs and increase their productivity in ways that far exceeds simple bug removal.

This is a key point. It’s software developers that build software and if we’re just finding bugs and assisting their removal, no real lasting value is created. If we take our job seriously enough we’ll ensure the way we go about it creates real and lasting improvement. Making developers better, helping them understand failures and the factors that cause them will mean fewer bugs to find in the future. Testers are quality gurus and that means teaching those responsible for anti-quality what they are doing wrong and where they could improve.

Here’s Tony’s exact words:

“The real value of tests is not that they detect bugs in the code,

but that they detect inadequacies in the methods, concentration

and skill of those who design and produce the code.”

 – Tony Hoare 1996

Now replace the word “tests” with “testers” and you end up with a recipe for your career. I imagine I’ll be examining this subject more in future posts. Follow the link above to get Jim Larus’ take on this as well as a guided tour through some of MSRs test technology, some of which is wide of Tony’s mark and some a bit closer.