Test Estimation

I just read a post from Shrini on test estimation, and it triggered a thought that was tangential enough that I thought it made sense to try to capture it here rather than as a reply to Shrini.

Numerous writings exist on estimation for programmers / developers, and many testers have similarly written about test estimation. I think that the fundamental problem with "test estimation" is that it shouldn't exist - at least in isolation from other estimations.

The problem is that test estimation models need to be largely based on the development estimation models they are coupled with. For example, if a development team writes crappy code and gives it to the test team as soon as it compiles so they can "test the bugs out of it", the development estimation times will be low, but the test estimation times will be relatively high. On the other hand, if the development team unit tests the heck out of their code, and are diligent running static analysis tools and in code review, the test estimate may be lower.

The answer, of course, is that there should be one estimate. What is the estimate to be "done". The teams that use scrum correctly already know this. They estimate when a component, feature, or product will be "done", and they carefully define "done" to include both development and testing activities. Frankly, I don't see how else you can estimate accurately without defining "done" regardless of your development approach.

How common is this scenario (probably not too hard for most of you):

Manager: Bob - how long will it take you to code this feature?

Bob: 'bout two weeks.

Manager: Great. Frank - how long will it take to test?

Frank: Two weeks sounds good

Manager: Nah, you can do it in one.

-- 3 weeks pass --

Manager: Frank - are you done testing yet?

Frank: Not even close. Bob checked in his code on time, but it was barely working, and I've spent most of the last week blocked while Bob is fixing the first 50 bugs I sent him.

Rather than an estimate of 2 weeks for development, and 1 week for testing, this component could have had an estimate of 3 weeks to "done"++, and Frank and Bob should work together to both come up with that estimate and deliver on time or adjust the initial estimate. Otherwise, Bob won't consider testing implications or code quality in his estimates, and without that knowledge, Frank's estimate is nothing more than a SWAG.

I'll go out on a limb and say this: Stop doing test estimations. They're nothing more than guesses used to put buffer into the schedule**. Instead, estimate how long it will take for implementation and testing combined, then have testers and developers work together to deliver on time.

++ Note that scrum (done correctly) would encourage breaking this estimate down into pieces no longer than 2 days.

** There will inevitably be some time set aside in the schedule for integration and acceptance testing. My comment above is in regards to feature or component estimates.

Comments (5)
  1. Shrini says:

    I agree with your point that test estimation can not be done in isolation with development estimates….

    However, to certain extent, test estimation should work on its own – especially in IT outsourced environment. Here, a testing service vendor is brought in as "quality police" and such cases, dev and test work with a "wall" in between them. This is unlike contexts like Microsoft, where dev and test work together (so is true in agile environments).

    So question would be "How long it takes to complete testing?".

    Among several parameters a tester needs to consider for estimation – dev estimate, their code maturity are key inputs.

    your suggestion of "stop doing test estimation" is BOLD one but consider those environments, where dev and test work work seperately and as such require two separate estimates. In some cases, both dev and test barely talk to each other …

    But,  this post of yours have really given some interesting things to consider in my "mission – test estimation model".

    Most importantly – development and testing models.

    So If I were Frank – I would say "Depending upon quality of the code that  Bob delivers, I take anywhere between 1 week to 5 weeks".

    Using some scrum principles is a good idea …. "Make sure to keep your units of time as small as possible – bigger a single unit of work, higher are the chances of errors in estimation"


  2. Alan Page says:

    I thought about outsourced testing, but didn’t address it as I thought it would be distracting to my point (so thanks for bringing it up).

    I do stand behind the principle that estimating software testing in isolation from dev estimation is impossible, so in a way, you’re asking how to do the impossible. You could give the 1-5 week answer above, but we know that option wouldn’t fly in the real world.

    I have two suggestions – both have their pros and cons, but may help with estimation in this scenario. I’ll start by reiterating that test estimation in a vacuum cannot be done – i.e. if I’m asked "We’re writing calendar application for meeting room scheduling – how long will it take to test", my answer will always be "I have no clue". However, if I were to find myself in that situation, here’s what I’d do.

    Option A:

    This one makes the most sense to me. When I need work done on my house, I call a few contractors, they come talk to me about what I want done, then they measure things and look up things and poke around a bit, and finally they get back to me with an estimate – both in time and money. I think outsourced testing could do the same thing. Want me to test your application? Sure – send it over, I’ll have our chief test estimator spend some time (length depending on size of the app) with the app to do an initial assessment, then I’ll get back to you with an estimate of testing time.

    In fact, some of the contractors I’ve used have given me multiple estimates – for example, one each for the "basic" job, and one or two others with upgrades such as a jetted tub or higher quality windows. Test estimation could do the same thing. My estimate could be "1 week to find most of the critical issues that your we expect that your customers would run into. In two weeks we can also offer reports with a breakdown of the top issues by area and a risk analysis report."

    But without seeing the software, I wouldn’t give an estimate.

    Option B:

    Not as good as option A, but this will work if the dev shop doesn’t trust your estimation. In this case, the dev shop employs at least 1 test minded person. They don’t do much testing, but they do work closely with every developer on defining "done" for a component and train developers to come up with accurate test estimates. These test estimates are presented to the outsourcing company along with the app to test – "Here’s our app – our data says it will take 2 weeks to test – thanks".

    Although this could work in theory, most outsourced test teams I work with don’t want to be told how long something takes to test. They may even go out of their way to prove the estimate wrong. I’d also bet that it would take a few iterations until the test estimates accurately reflected the time spent by the oursourcing team – keeping in mind that the test estimate in this scenario is not reflective of the people doing the actual testing.

    But it could work.

  3. Much as I sometimes wish I could say, "oh, I can’t do a test estimate", that one doesn’t really fly in my real world, either! We’ve hit on a pretty good compromise at work, though.

    The key to it all is that we don’t estimate stories (dev or QA) until there are defined acceptance criteria. (I should note that we use a variant of Extreme Programming as our development process, and one of the things we’re very strict about is describing acceptance criteria as we’re describing requirements.) Then estimation is done by both development and QA – so you get one estimate for both. The development estimate accounts for building the feature and building the tests that prove the feature works; dev is "done" when the automated acceptance tests pass. The test estimate accounts for any manual tests involved and integration testing (based on the size of the feature and its relationship to the rest of the product). These are then combined into a single estimate for the entire implementation, and that is what’s presented to product management et al.

    The biggest problem with this is that there is still a "gut feel" portion of the estimate, based on two things: (1) how buggy is the feature going to be; and (2) how much integration testing is necessary before we are comfortable with the feature?

    So, handling of the "how buggy is this feature" is done basically by folding it into the development portion of the estimate. There may be a lot of bugs in other areas of the code (whoops!), but that feature at least won’t spend a lot of its "testing" time waiting for development to fix bugs. This hinges on having thorough acceptance criteria, but practice gets you good at that one.

    Handling of the "how much integration testing is necessary" is much more of a dark art, and this one we could use more practice with. Our general criteria is to weight our basic estimate by size of the module, interaction with external programs, and interaction with other modules of the overall product; this tells us the likelihood that the code will wind up back in dev fixing something and how long it’s likely to be back in dev. But this one still boils down to a gut feel, and we’re definitely looking for a better way.

  4. Shrini says:

    Estimating stories … Thanks Catherine for bring agile part of story(!!!) to this discussion …

    Let me think … if one were to ask "Here is a story – that is about creating a form that takes email address of a person and add his/her to mailing list and sends mail with registration information" ..

    how much time you take to test this story …

    That is like talking about a set of test cases …

    Depending upon the size of the story, you can get pretty close in estimating in terms of stories … when we start adding few estimates together ..especially when the stories are linked… things start becoming complex …


  5. Test #5 of the Test Test asks if the development team values the test team. Catherine pointed out that

Comments are closed.

Skip to main content