ClassCleanup May Run Later Than You Think

In addition to the well-known TestInitializeAttribute, the Visual Studio Team System unit testing framework also includes both initialization and clean-up attributes for both individual test cases, classes and assemblies. Although the use of these attributes may seem straightforward, some methods decorated with these attributes may not run in the order you would intuitively think (or maybe my intuition is just weird). Let’s do a little quiz! Consider these two test classes:

public class TestClass1
    public TestClass1() { }
    public static void InitializeAssembly(TestContext ctx)
    { Debug.WriteLine(“AssemblyInitialize”); }
    public static void CleanupAssembly()
    { Debug.WriteLine(“AssemblyCleanup”); }
    public static void InitializeClass(TestContext ctx)
    { Debug.WriteLine(“TestClass1: ClassInitialize”); }
    public static void CleanupClass()
    { Debug.WriteLine(“TestClass1: ClassCleanup”); }
    public void InitializeTest()
    { Debug.WriteLine(“TestClass1: TestInitialize”); }
    public void CleanupTest()
    { Debug.WriteLine(“TestClass1: TestCleanup”); }
    public void MyTestCase1()
    { Debug.WriteLine(“TestClass1: MyTestCase1”); }
public class TestClass2
    public TestClass2() { }
    public static void InitializeClass(TestContext ctx)
    { Debug.WriteLine(“TestClass2: ClassInitialize”); }
    public static void CleanupClass()
    { Debug.WriteLine(“TestClass2: ClassCleanup”); }
    public void InitializeTest()
    { Debug.WriteLine(“TestClass2: TestInitialize”); }
    public void CleanupTest()
    { Debug.WriteLine(“TestClass2: TestCleanup”); }
    public void MyTestCase2()
    { Debug.WriteLine(“TestClass2: MyTestCase2”); }

There are two test classes that are basically identical. The only real difference is that TestClass1 hosts the AssemblyInitialize and AssemblyCleanup methods in addition to the other methods, which basically all just write to the debug window that they are executing. What will be the output if you run all (both) these tests in debug mode?

As you may have guessed from the headline, the tricky part is ClassCleanup. Before I began to think about it, I just assumed that the test run would spin up TestClass1, execute the tests in that class and clean it up, then proceed with TestClass2 in the same manner.

Obviously, I was forgetting that unit tests are not ordered. Although they are executed sequentially, the order in which they are executed is not guaranteed. This is, in fact, a good thing, because it helps you remember that test cases should always be independent.

In any case, here’s the result from my Output Window:

TestClass1: ClassInitialize
TestClass1: TestInitialize
TestClass1: MyTestCase1
TestClass1: TestCleanup
TestClass2: ClassInitialize
TestClass2: TestInitialize
TestClass2: MyTestCase2
TestClass2: TestCleanup
TestClass1: ClassCleanup
TestClass2: ClassCleanup

It’s hopefully no surprise that AssemblyInitialize runs first, and AssemblyCleanup runs last. Notice, however, that although TestClass2’s ClassInitialize execution is deferred until needed, this doesn’t mean that TestClass1’s ClassCleanup executes immediately after the last test case in the class! In fact, it waits until all test cases are executed, and the executes together with TestClass2’s ClassCleanup.

This surprised me at first, but that was obviously only because I hadn’t really thought it through: Since tests are, in principle, unordered, there’s not guarantee that all tests in TestClass1 are executed in immediate succession. Theoretically, the execution engine may pick a test case from TestClass1, then one from TestClass2, then another from TestClass1, etc. Since that is the case, there’s no guarantee that all tests from one test class have been executed before a new test class is initialized, and thusly, all ClassCleanup methods may as well be deferred until all test cases have been executed.

When this first surprised me, I had written some code in one class’ ClassCleanup and expected it to run before another class’ ClassInitialize. What I got instead was a strange interdependence bug, where all tests in both classes succeeded when executed together with other tests from their own class, but as soon as I ran all tests from both classes, the tests in the second class failed. The lesson here is that not only should you treat your test cases as independent, but the same goes for initialization and clean-up code.

Comments (6)

  1. Guy Dalberto says:

    Nice post, but I think you do not reach the correct conclusion.

    Your assumptions were correct, it’s the VSTS implementation that is wrong.

    Fast simplistic Unit Tests do not need parallel processing, and complex slow ones (Database, external Process, ..) cannot be independent if thery use some common ressources, so VSTS should be predictive, not randomistic.

  2. ploeh says:

    Hi Guy

    Thank you for your comment.

    The main point of this post was to share a finding on how MSTest works, and I believe I arrive at the correct conclusion.

    MSTest isn’t random. As far as I can tell, the algorith is completely deterministic – it’s just different that you might first expect.

    My point about MSTest giving no guarantees about test ordering is, AFAIK, true for all the xUnit testing frameworks. It’s by design, since you should never design your test to depend on executing in a specific order.

  3. Interesting,

    I really learned how MSTest works!

    Keep up the good work

  4. bo says:

    I definitely don't like the way MSTest is implemented in this regard.  I have tests that have to install, start, and then tear down windows services.  This dependency exists for multiple classes in the test project.  

    Generally, my idea has been:

    ClassInit – Install the windows service

    TestInit – Start the service

    RunTest with dependency on service

    TestCleanup – Stop the service

    ClassCleanup – Uninstall the service

    Sometimes I get errors that a service couldn't be started because it was already running, etc. etc.  This should NOT be possible if the code ran in a sensible order!  All tests pass successfully if I run them 1 class at a time rather than the whole solution together.  LAME!

    As other unrelated tests get added to the solution which tests fail can change just because the order of the test runs changes (seems pretty random to me!  I HATE that!).

    Now I have to redesign the whole structure of the tests it seems – though I'm still trying to get my mind around everything that is going on.

  5. Jostein Kjønigsen says:

    What this results in for all means and purposes is that ClassInitialize and ClassCleanup has no useful purpose since order cannot be assumed to be deterministic anyway.

    It's only function at this point is -misleading- programmers into thinking they have a deterministic execution structure when they don't have it, and thus introduce bugs you wouldn't otherwise get.

    TFS would in fact be a better product with these highly misleading attributes completely absent. Not calling this behaviour a bug is incomprehensible.

  6. Evgeni Raikhel says:

    I totally agree with the above conclusions by Jostein Kjønigsen:

    I case you run a batch of tests and at some point a single test fails, there is no way to release its resources (as the CleanUp routine will be deferred). So all test that follow on will fail as well as they don't have access to those resources.

    It is actually a bug – just like allocating memory and not releasing it on exit.

    Just a shame that I'm writing this post almost three years after the previous one and nothing has changed…