Software Statistics Engineer?

I recently read an (internal, so I can't give a link, sorry) article by a developer pundit. In the article he opines that while the current practices of unit testing, etc. are having a definite impact on code quality, eventually we will reach a point where it becomes much harder to find bugs, at which point testing becomes a lot harder.

He sees Test as having three responsibilities:

  • Finding bugs that devs miss.
  • Running in the customer's environment.
  • Analyzing quality metrics.

(Note that this is his definition, not mine, so don't yell at me if you think it's all wrong. I'm not entirely happy with it either.)

Clearly the first of these becomes a lot harder as the code gets cleaner. My team is looking hard for new metrics to judge the quality of our code and our testing because metrics such as test case count and number of bugs found just aren't interesting anymore. (Yes, I know they weren't really ever interesting. Humor me here.) If you test your specs rather than waiting for code to be delivered before you say "Hey -- what exactly should happen when I twist this widget", and if your developers thoroughly test the code that they write and so find most of their bugs before ever checking in, then naturally there will be a lot fewer bugs for Test to find. This is A Good Thing.

The second role isn't affected at all. Although I disagree this is just Test's responsibility. Just like the entire feature team is responsible for ensuring the quality of their feature, the entire team is responsible for knowing how the customer will use the feature, which includes understanding what the customer environment is and running within that environment. Dev is not doing enough testing if they aren't running in the customer environment at least some of the time. I do however agree that having more time to focus on interfeature behavior because Dev lets fewer bugs through is (again) A Good Thing.

Which brings us to what this pundit sees as Test's third responsibility: analyzing quality metrics. He believes that just following disciplined development practices will only take one so far, and that breaking through to the next level of development efficiency will require devs to collect data throughout the process and use that data to accurately determine when all the bugs have been eliminated. He seems to think that we testers are "salivating" at the prospect of having all this data to fold, spindle, and mutilate into (I guess) oodles of test cases.

At first I had a violent reaction to this. "I'm a tester, Jim, not a statistician!" Subsequent readings have mollified me somewhat; he's not suggesting that the *only* thing tomorrow's testers will do is crunch numbers, just that we will be doing more testing-once-removed (as it were), sifting and sorting all this information to spot trends, identify weak areas, and find ways to make life better for us and our team and our product. I don't have any ambitions around becoming a statistics professor, but I certainly wouldn't mind more data to help guide my testing.

But then he goes and says "[T]est becomes quality assurance". No no no no no! I'm a tester, Jim, not a QA engineer! Quality is the entire team's responsibility.

[Thinking calming thoughts....]

Oh, I get it: he thinks that quality assurance is going to become so important that we will have an entire team dedicated to pursuing it.

I agree, I guess. It's just that I think we already have that team: the entire product team. We have a lot to learn about en/assuring quality, certainly, but we are the ones who must learn it. Doing anything else is just passing the buck.

[Postscript: One of the cool things about Microsoft is that anyone can fire off an email to anyone else -- regardless of how far apart the two people are in the org chart -- and be likely to get a reply. Case in point: I sent this post to the abovementioned pundit asking for his comments on my comments on his comments. He replied that "the tester responsibilities I listed were never meant to be exclusively for testers" but rather that the entire feature team (including even y'all who sign up to be beta testers!) should be doing these things. He focused on testers because he believes somebody needs to be the point person for doing this and he thinks Test is the logical group to do that.

So maybe we aren't as far apart in our beliefs as I thought. I have an ally!  <g/>]

*** Comments, questions, feedback? Want a fun job on a great team? Send two coding samples and an explanation of why you chose them, and of course your resume, to me at michhu at microsoft dot com. I need a senior tester and my team needs program managers. Great coding skills required for all positions.