To cleanup or not to cleanup

One attribute typically considered important for automated test cases (or almost any other test case for that matter) is cleanup.

The test shall leave the system in the same state it was in before the test ran.

This is a good theory, but I wonder how practical it is. Anything a test does has potential to have some impact on the system I am unaware of. Even if I close the application I was testing, remove any files my test may have created, and remove any other artifacts, the fact that some amount of code ran leaves the system in an unknown state. Can you really leave the system in the state it was in before the test ran?

When I was on the CE team, we just threw this rule out the window. Every test suite (definition in this context == collection of tests for testing a particular functionality or attribute as defined by the tester) ran from the exact same starting point - because we automatically re-flashed a new OS onto the device before every test suite. Flashing a new OS took a minute or two, but guaranteed a known environment at the beginning of each test run. Short of a solution like this, I don't know how this rule could be accurately observed.

To be clear, I'm not saying I liked this approach. One advantage of not restoring the system to a known state is the exact scenario that this rule tries to avoid. Running code causes the system to change. One of the things I always worried about was "What if a test is causing an OS memory leak or corruption or some other badness". What if the badness is severe, but just not severe enough to cause crash in the tests? The system may be in an awful state after the tests complete, but we'll never know, because we throw away the system state after the test executes" (yes, you could run a memory check or other diagnostics after tests run and maybe catch stuff like this...).

The point is, that I don't know if the goal of cleanup is attainable, or even if it is something desirable. Sure, if you create a thousand files, maybe you need to clean them up, but maybe we need to embrace change rather than trick ourselves into avoiding it.

Let me put it this way: Forget clean up. Don't bother. Unless it's necessary to clean something up, leave the system in the dirtiest state you can. Heck - it's a lot closer to the way customers run the software anyway. My parents never delete files, and they certainly don't clean up reg keys. I just can't think of many good reasons to put effort into cleaning up after tests. Of course, you are welcome to let me know if I've forgotten something.