What's your test automation strategy?

I overheard some test managers discussing problems with their test automation effort, so I couldn't refrain from asking the redundant question, "What is your test automation strategy?" They looked at me as if I had just beamed down from another planet and said, "c'mon, you know our strategy is to automate everything!"

It is unfortunately true that some managers drink the proverbial kool-aid and blindly regurgitate the 100% automation mantra or similar incantations such as "no manual testing" popular among agile pundits like Lisa Crispin.

Let me be clear. A goal of 100% automation is not a test strategy; it is a fantasy! Similar to the Disney fairytales where fairy dust causes magical transformations, evil is defeated, the prince marries the maiden, and everyone lives happily ever after forever automating everything is not practical or realistic. 

Perhaps the single biggest problem with most test automation efforts is lack of a practical strategy. A practical test automation strategy is one that provides a pragmatic solution to address specific business needs with well-defined, measurable goals based upon realistic expectations.

Business needs drive a lot of the change in any organization, and usually involve cost saving measures, quality improvement, or increased customer satisfaction. A business need for test automation includes reduced testing time. (This doesn’t mean reduced ship cycles; it simply means the time it takes to perform certain tests during the product life cycle can be shortened.) For example, the Build Verification Test (BVT) is a necessary test suite to verify the stability of each new build. Depending on the size and complexity of the product a manual BVT suite can be very time consuming. An automated BVT suite (which should be 100% automated including results validation because it establishes a baseline measure on build stability and the tests remain relatively static over the duration of the development life cycle) can substantially reduce the time spent in this phase of testing especially in iterative build environments where the team is getting daily or even weekly builds. It doesn’t take long to realize the cost savings over the product life cycle.

Test automation strategies must also have realistic expectations. For example, I have never been convinced that finding “new’ bugs is a realistic expectation for test automation. (Yes, it will occasionally find some new bugs, but let’s face it…the majority of the 5 -15% of the bugs exposed by test automation in production environments are regressions.) I have never seen data that suggests increased automation reduces the overall development cycle. Nor will test automation eliminate testers. (This is a false hope imagined only by prima donna developers and bean counting managers scheming of ways to find value in their Masters in Business Mismanagement degrees.) So, what are realistic expectations for test automation? Well, I can reasonably expect test automation to identify stress issues such as mean time to failure (MTTF) and mean time between failures (MTBF). I can reasonably require test automation to establish baseline measures such as BVT suites or regression suites. Test automation is a pragmatic solution for load any type of load testing or other forms of concurrency testing.

Finally, a good test automation strategy must have measurable goals so we clearly understand what success looks like (or identifies where we need to improve). Without goals we are developing automated tests just to say we are automating. Unfortunately, I occasionally see teams with goals of automating n% of existing tests. This really doesn’t make much sense because it doesn’t take into account logical decisions of what tests should be automated (remember, not all tests need to be or should be automated), so some redundant tests or run-once type tests are automated (which may not be the best use of your limited testing resources). Also, the ‘existing’ set of tests is usually a moving target, so that means the goal is a moving target, which means we can never achieve the goal. Goals for test automation should be specific, measurable, achievable, realistic, and timely (SMART). Set short term and long range SMART goals for your test automation effort. For example, a short term goals might be 100% automation of the BVT suite within 1 week after the first build drop. Long term goals might include design elements and processes to transfer automation to sustained engineering or maintenance teams, or 100% language neutral automation that will execute on any localized (or pseudo localized) language version.

Test automation is expensive. Testers have a lot of work to do in a very limited timeframe, so it is important that we use our testing resources effectively. A well defined automation strategy will establish clear goals, set expectations, and provide practical, automated solutions.