Pesticide Paradox

In this article I will be attempting to explain one of the biggest (or what I feel is the biggest) drawback of traditional automation testing – the phenomenon of product code becoming more resistant to the test code and resulting in less bugs being found over time by the same test code. This was originally stated by Beizer as: “Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual. ” We can consider this to be similar to actual pesticides becoming less effective with time since the pathogens they attack develop resistant strains (if you consider the software being tested to be “evolving” because of the fixes being added).

Nothing gets the point across better than this quote of a quote from Intelligent Test Automation:

“Highly repeatable testing can actually minimize the chance of discovering all the important problems, for the same reason that stepping in someone else’s footprints minimizes the chance of being blown up by a land mine. ” - James Bach

For starters, by automation testing, I simply mean a bunch (or suite) of tests which can be kicked off together and which require no manual intervention for running the tests or validating the results of each of the tests. Thus once kicked off, an automation “run” simply gives you a summary of which tests passed and which failed. Very nice. Very convenient. And very static.

For the most part when you code up a test (which is equivalent to manually running a test case for the first time) you will either find a bug in the functionality being tested, or the functionality will be working as expected. This moment is pivotal and the key insight here is that if you do not find a bug at this point, your test code will likely never find another bug in its lifetime. It’s in a way dead test code. To be fair, it does serve one important purpose – that of regression testing – which refers to the testing of new code for side effects that break existing functionality. For e.g. you implement a feature to export data to XML and indirectly cause a “regression” which breaks existing import functionality. Testing for the new export functionality will not be enough to catch this bug – you need to validate all existing functionality in other features and that is where automated tests come in most handy.

But still – the chances of the test you just coded up finding any bugs after its first shot are pretty slim. Does that mean there are no more bugs remaining in your feature? Probably not if the feature we are talking about is sufficiently complex. It simply means that what you tested for did not have bugs. And there is only so much you test when manually coming up with individual test cases.

So in a sense as you keep writing test code and finding the basic bugs in the functionality, the product code keeps getting more “resistant” to your test code. So early in the project cycle the test code will typically be finding more bugs but a little later the bug rates go down as the product code contains all the fixes for bugs found so far and start to fare better against the test code. This is typically perceived as natural and expected and in many cases it actually does translate to a better product, but it can also result in a misleading sense of quality in the product being shipped. Why? Because all that has happened is that the cases you have coded up have been validated. If the system under test is fairly complicated so that certain (or in the worst case, many) scenarios could not be covered, then you got yourself a false sense of good quality.

The mitigation here would be to either add lots of new cases which would cover all or most of the scenarios where bugs might be present, or to adopt a methodology which will allow you to reuse the test code you have written to automatically test new scenarios and code paths. The key take away is that there should be no coding required on a per test case basis since that really slows you down and test case generation should be automated so that you eliminate the manual intervention required for the actual listing of a new case. In essence we want a machine to be able to create new test cases and then run those test cases and validate behavior of the system. And that, my friends, brings us to what I wish to sell – Model Based Testing. I pray for the pathogens (read product code)!

To be fair – I do not wish to portray model based testing as a panacea for all testing pains. And in a later post I will try and look at the shortcomings and concerns with this approach – but for now lets take in this one simple, and often, inconspicuous shortcoming of the way we currently test most software.