test case reuse (in the future)

I’ve given my ‘future of testing’ talk four times (!) this week and by far the part that generates the most questions is when I prophesize about test case reuse. Given that I answered it differently all four times (sigh), I want to use this space to clarify my own thinking and to add some specifics.

Here’s the scenario: one tester writes a set of test cases and automates them so that she can run them over and over again. They are good test cases so you decide to run them as well. However, when you do run them, you find they won’t work on your machine. Your tester friend used automation APIs that you don’t have installed on your computer and scripting libraries that you don’t have either. The problem with porting test cases across machine boundaries is that they are too specific to their environment.

In the future we will solve this problem with a concept I call environment-carrying tests (nod to Brent Jensen). Test cases of the future will be written in such a way that they will encapsulate their environment needs within the test case using virtualization. Test cases will be written within virtual capsules that embed all the necessary environmental dependencies so that the test case can run on whatever machine you need it to run on.

The scope of technological advances we need for this to happen are fairly modest. However, the Achilles heel of reuse has never been technological so much as economic. The real work required to reuse software artifacts has always been on the consumer of the reused artifact and not on its producer. What we need is an incentive for testers to write reusable test cases. So, what if we create a “Testipedia” that stored test cases and paid the contributing tester, or their organization, for contributions? What is a test case worth? A dollar? Ten dollars? More? Clearly they have value and a database full of them would have enough value that a business could be created to host the database and resell or lease test cases on an as-needed basis. The more worthy a test case, the higher its value and testers would be incentivized to contribute.

Reusable test cases will have enough intrinsic value that a market for test case converters would likely emerge so that entire libraries of tests could be provided as a service or licensed as a product.

But this is only part of the solution. Having test cases that can be run in any environment is helpful, but we still need test cases that apply to the application we want to test. As it turns out, I have an opinion on this and I’ll blog about it next.

Comments (8)

  1. radi.v says:

    Using virtualization for porting test cases from one environment into another is a good thought. However, IMHO this might leave a room for false positives popping up based not on the actual software under test failing, but the virtualization component failing to adapt properly to the test environment. As a result, it could force the tester to troubleshoot something that is outside of the scope of the testing cycle.

  2. Guy kolbis says:

    Have you ever though to sell you test cases? I am pretty sure that most of you have not. This is not

  3. pberry says:

    I share your optimism to a certain degree but my take on testing reuse is more guarded.

    Just as software/code reuse can never be made to truly work, since every software project is necessarily different and specific, so I don’t see how testing, automated or manual, could ever be portable either.

    You can certainly generalise testing techniques, strategies, etc, depending on the type of project under test: stand-alone application, public web site, corporate intranet, etc, but the devil is in the details. Testers could take direction from a generalised set of peer-reviewed test cases but the onus would still be on them to select the most appropriate ones, put them in the right order, and work out the details in order to get a decent test strategy together.

    Still, food for thought…

  4. lightbody says:

    I just blogged my thoughts on this post here:


    Basically, I see this vision of the future as very much possible, but it depends on it being *easy* to snapshot virtualized state (and not all state is easy to capture) and it being *fast* to re-created said states when running test suites.

    I think the speed issue is mostly solved, thanks to EC2 and other utility cloud computing projects. But the ease-of-use issue is a long ways off, and I wonder if it can ever be solved properly. A lot of taking proper application state snapshots involves application-specific knowledge that may never be able to be generalized.

    And even if taking snapshots is easy, some parts will always have to be mocked out (ie: dependency on real-time stock quotes, etc). Hopefully this is a hurdle that the industry can jump, because if it can this approach could drastically change how software is built.

  5. SRLTeam says:

    How many times we can see similarities in different solutions comes from different companies? The answer

  6. Toddsa says:

    Future testing strategies need tools to identify and resolve problems during code execution. In order to do this though we need to alter our execution environments.

    But for test cases we don’t need to wait for the future in order to make them better. Test cases will be reused if they are useful, simple, descriptive and discoverable.

    Writing test cases for reuse is easy and actually have people use them is easy. Just follow some guidelines and you should be fine.

    1) Discoverable:

    Test cases should easily be found and ideally checked in next to the dev code. When making a change to code if a developer has to look around in databases and .xls sheets for cases believe me no tests will be ran. But if the cases are checked into the same branch as the dev code it increases the chance of reuse. It is even better if the build environment runs the cases before allowing check in.

    2) Simple:

    The test case needs to be simple. I should be able to find the executable and run it and have it report pass or fail. I should not need a database or figure out exactly what properties that need to be set in order to run the case. If I can’t run the .exe and it work, then the case will never be ran. The test case should have simple logic and if you need to pass complex data to it than it should be multiple test cases. For example you should have a separate test case for each permutation of CreateFile(…) and not rely upon some input variable for conditional logic. More generically said, no direct input from the user should alter the test case execution path.

    3) Environment:

    There is a lot of discussion about having the environment follow the test case or the test case setting up the environment. For some scenarios this can be good but for most areas it is a waste of time to attempt. The test case should check for environment requirements and report them when they are not met but not act on them. How often is a test case ran outside of a test environment anyway, and if it is, would that be supported? If you are going to run test cases you know to install, configure or setup what you are testing. Adding this logic to the test case complicates it and makes it so you cannot easily reuse the case, just don’t do it. And as for virtualization environments, if you are not testing virtualization, do NOT test in virtualization environments. If you do, have fun debugging for many hours.

    This is kind of going off on a rant, but basically to improve testing you don’t need to add in fancy test databases, execution environments or virtualization. Just make it simple and easy for others to use. One verification for writing test cases to gauge how you are doing is see if you can send your test cases as a .exe outside to a vendor and if they can execute them without dependencies or asking you follow-up questions then you are on the right track. Testers make it too complicated some times when it really is not. We tend to waste too much time on buzzwords for our reviews, instead of just working on testing.

  7. [Nacsa Sándor, 2009. január 13. – február 3.]  A minőségbiztosítás kérdésköre szinte alig ismert

  8. [ Nacsa Sándor , 2009. február 6.] Ez a Team System változat a webalkalmazások és –szolgáltatások teszteléséhez