How many tests are enough?

How many
tests are enough?  How much code coverage is enough?  How relevant are the
number of tests, code coverage numbers and bug counts in knowing when testing
is done?

Let’s
say, for example, that I’m testing a 5000 line component and I’ve written 2000
tests that hit 90% of the code blocks, and I haven’t found a bug in a week.  Am
I done?  Of course, the answer is “maybe”, “heck if I know”, or “it depends
what else you’ve done”.  Coverage numbers just mean that my tests have hit the
code, but don’t tell me anything about the data that’s been used in those code
paths. The number of test cases is irrelevant, as is the incoming bug rate.  If
I keep running the same tests on the same code, I’m probably not going to find
new bugs.  There could be ship-stopper bugs buried in the 10% of the code I
haven’t tested just as easily as in the 90% I have tested.

One
answer would be to look at the use-cases or customer scenarios that accompany
the feature.  Chances are that if all of these scenarios work perfectly, that
customers won’t hit the remaining bugs.  This, of course assumes you have
use-cases…and that they were written with the customer in mind.  If this wasn’t
the case, you have bigger problems anyway.

I’d also
test with fault injection (I’ll leave FI methods to another post).  Chances are
that the 10% of the code I haven’t hit are in error paths, and that I haven’t
fully exercised error paths in the other 90%.

Model
based testing may also help me out here, but so could dozens of other
techniques.  In a ship date driven world, which techniques do I use to have the
best odds of shipping defect free software?

What
else do you do to know you’re done?  What metrics do you care about in this
situation?  Is it possible that I actually am done, and that I’m “over-testing”?