Test Cases Are Maintenance Hogs

These problems all combine to make test cases a maintenance nightmare. Anytime the application under test changes – regardless of whether the UI changed a tiny bit or an entire feature was dramatically revamped – every affected test case must be updated. Identifying these test cases can be quite difficult since every test case likely does things slightly differently. One test case may know the exact location of the widgets it presses while another determines the locations as it executes. Some test cases may get data directly inline to their verification checks while other test cases may push the data into temporary variables. Some test cases may be annotated with the feature they cover, but inevitably some test cases will be annotated incorrectly or not at all. Test case names and organization can be used to guide this effort but as with annotations these will not be correct in every case.  Consequently, changes effectively require every test case to be reviewed. Then you must run the full set of test cases to discover which ones you missed.

While code coverage and test prioritization tools can simplify the process of finding the test cases that must be updated, the maintenance must still be carried out by manually reviewing and editing each test case individually – no easy task either. For example, if a widget changed from a check box to a pair of push buttons, all calls to check the box have to be changed to push one button while all calls to uncheck the box must be changed to push the other button. If application functionality has changed you must now actually understand what each test case is doing, identify the probably multiple lines to rip out, and then replace them with multiple other instructions. This painstaking process must be carried out for each subsequent test case as well, as it may be doing execution or verification slightly differently, which difference may or may not matter.

Thus a change that may take the developer ten minutes total (including running the full checkin suite) can result in days of work for the tester. The more test cases Test writes the more work each change potentially requires. In fact, drastic changes can often be easier to cope with than small changes because deleting entire sets of test cases that are now invalid and replacing them with brand new test cases often takes much less work! This is a strange situation; after all, automated tests are supposed to save time, not use it up.

These problems are only exacerbated by the longevity test cases must have today. Test cases for new features are clearly necessary in order to ship those features, but they are just as important during subsequent releases when they help ensure their features continue to work correctly. Further, test cases are a critical part of the sustained engineering process that continues even after the product is no longer being actively developed. Reliable test cases that are simple to understand and easy to maintain are vital to the efficacy of this effort.

Our technologies localize maintenance to single points of change within our automation stack, allowing test cases to be nearly maintenance-free. (Except of course when reacting to significant changes in product functionality. One could imagine abstracting a product’s functionality such that even major changes in behavior are absorbed somewhere in the library, but we believe this is taking things rather too far.)