One More Hallmark

Darren writes:

Hi, I read through your list of attributes today with interest.  One of the attributes that interests me but, that you did not mention was:  Ability to provide accurate estimates. 

I'd be interested to read *your* take on that attribute.

Estimation is indeed as important a skill for testers as it is for developers.  It has been my experience however that most Test teams don't even attempt to estimate the work they have to do. 

The more cynical testers will tell you that there's no point in estimating testing work since the product is going to ship whenever management decides it's ready regardless of how much testing there is left to do.  (Umm...if you're in this position, come work for me.  I need testers!)  I disagree.

Others say that since you can never be completely done testing (I agree; there are always more bugs to be found) there's no point in estimating those never-ending tasks.  I disagree.

Yet others say that testing is essentially an ad-hoc interrupt-driven process and thus can't be estimated.  I disagree.

Testing is not the amorphous, impossible-to-estimate activity it's often made out to be.  Rather, testing is a cycle of:

  • Getting to know the feature/product/API to be tested-how it works, what it's supposed to do, how it's supposed to interact with the rest of the world.
  • Brainstorming ideas for how "supposed to" might turn into "oops". 
  • Writing and executing tests to verify that "supposed to" happens and "oops" doesn't. 
  • Logging bugs. 

This generally isn't a "do step A, then step B, then..." process, of course.  Everything mushes into everything else.  You may need to play with the feature to understand "supposed to"; while doing so you may stumble across bugs, which you log; investigating a bug often triggers a series of "I wonder if" and "If that's happening then I bet" and "Ooohhh".  But the steps are all there.

Which means you can estimate them.

Trying to estimate these steps for an entire feature with any accuracy is pretty much possible.  Break them down into smaller steps, however, and estimation becomes much easier.  A feature isn't one big glob of goo but rather a bunch of smaller actions and use cases that together form the feature.  Estimate tasks for those smaller pieces.  If they are still too big to estimate with accuracy, break them down further yet.  I generally keep going until I'm down to tasks I think I can complete in about sixteen hours.  The occasional longer task is okay, but any task estimated forty hours or more is definitely too big and must be broken down into smaller components.

Estimation with any accuracy is hard.  The only way to get better is to practice.  Perfect estimates would be grand, but they aren't really necessary.  I tell my team I don't care how inaccurate their original estimates are;  their accuracy will improve over time.  What I care about is that they continually update their estimates so that we always know where we are now.

Anyone who trusts initial estimates is crazy anyway.  I don't remember where exactly, but I remember seeing some statistics that estimates provided at the start of a project are off by plus-or-minus fifty percent.  There's no way around this, really.  At that point it's too early to know what you don't know -- namely, the various problems that are going to crop up.  The closer you get to shipping the more you know and the better you can guesstimate how much work you really have left and how much time it's really going to take you to complete it.  It's only as you near the end that you really need accurate estimates anyway.

This however is most definitely not an excuse not to estimate your testing tasks.  Just be prepared for wide variation from your initial numbers.

Here's what we do:

  • Spend a lot of time in feature teams getting a good grasp on what exactly we're going to do this milestone and how we plan to do it.  This includes talking about not just the feature but discussing the tests as well.
  • Estimate how long all that work is going to take.  Again, this includes not just coding the feature but coding the tests and the automation stack to support it. 
  • Continually update those estimates throughout the course of implementing the feature.  This is the only way those initial, probably-way-off estimates will ever become useful.

Testing is very much part of a feature's total cost, so it's just as important to develop and maintain good estimates for testing the features as it is for coding the feature.  Testing may be a never-ending task, but estimates help you know how much longer "never-ending" will really take.


*** Comments, questions, feedback?  Want a fun job on a great team?  Send two coding samples and an explanation of why you chose them, and of course your resume, to me at michhu at microsoft dot com.  I need a senior tester and my team needs a dev lead and program managers.  Great coding skills required for all positions.

Comments (0)

Skip to main content