Re: When To Automate

Patrick comments on my When To Automate post:

I am confused by your third bullet in that unstable/newly developed/high change code should have automated tests written for it.

It has been my experience that test automation is a great deal of work to maintain and that the more changes there are to an application under test the more automation maintenance work is required. It almost never pays off to write automation against an application that is undergoing major changes to its UI, structure, or functionality. Having to rewrite large portions of the automation code and cases on every build often takes longer than it would take to perform those tests manually. Additionally it is of paramount importance that a human being’s intuition is leveraged on very new code to account for that which had not yet been considered. I have had very good luck using automation for regression testing or getting lots of boundary cases executed, but only on code that was pretty well established.

Am I missing something?

Anytime you believe automation to not be worth the trouble, I encourage you to change your viewpoint and find a way to reduce your automation cost. For example, although my application's user interface undergoes constant change we don't write tests directly against the UI and so our tests don't need to change. My application's functionality is changing on a regular basis as well, but because we write our tests in terms of user functionality rather than implementation details those changes don't affect our test cases either. (Learn more about our stack here and here). My developers are finding the gains they receive from writing unit tests more than make up for the minor pains they have in maintaining them.

That said, certainly there are cases where automation is not worth the trouble. (See my post I Want Testers, Not Automators for example.) And I completely agree that automation does not take the place of real live testers poking at code. My goal in writing automation is to free this real live tester from doing one set of test cases over and over and instead let me spend time banging on my app in new and devious ways. Of course, automation has all sorts of side benefits, such as big-time payoffs come sustained engineering time, and automatic checking for regressions. But freeing me from drudgery is the main point as I see it.