ClassCleanup May Run Later Than You Think

In addition to the well-known TestInitializeAttribute, the Visual Studio Team System unit testing framework also includes both initialization and clean-up attributes for both individual test cases, classes and assemblies. Although the use of these attributes may seem straightforward, some methods decorated with these attributes may not run in the order you would intuitively think (or maybe my intuition is just weird). Let's do a little quiz! Consider these two test classes:

 [TestClass]
 public class TestClass1
 {
     public TestClass1() { }
  
     [AssemblyInitialize]
     public static void InitializeAssembly(TestContext ctx)
     { Debug.WriteLine("AssemblyInitialize"); }
  
     [AssemblyCleanup]
     public static void CleanupAssembly()
     { Debug.WriteLine("AssemblyCleanup"); }
  
     [ClassInitialize]
     public static void InitializeClass(TestContext ctx)
     { Debug.WriteLine("TestClass1: ClassInitialize"); }
  
     [ClassCleanup]
     public static void CleanupClass()
     { Debug.WriteLine("TestClass1: ClassCleanup"); }
  
     [TestInitialize]
     public void InitializeTest()
     { Debug.WriteLine("TestClass1: TestInitialize"); }
  
     [TestCleanup]
     public void CleanupTest()
     { Debug.WriteLine("TestClass1: TestCleanup"); }
  
     [TestMethod]
     public void MyTestCase1()
     { Debug.WriteLine("TestClass1: MyTestCase1"); }
 }
  
 [TestClass]
public class TestClass2
{
    public TestClass2() { }
 
    [ClassInitialize]

    public static void InitializeClass(TestContext ctx)

    { Debug.WriteLine("TestClass2: ClassInitialize"); }
 
    [ClassCleanup]

    public static void CleanupClass()

    { Debug.WriteLine("TestClass2: ClassCleanup"); }
 
    [TestInitialize]

    public void InitializeTest()

    { Debug.WriteLine("TestClass2: TestInitialize"); }
 
    [TestCleanup]

    public void CleanupTest()

    { Debug.WriteLine("TestClass2: TestCleanup"); }
 
    [TestMethod]

    public void MyTestCase2()

    { Debug.WriteLine("TestClass2: MyTestCase2"); }
}

There are two test classes that are basically identical. The only real difference is that TestClass1 hosts the AssemblyInitialize and AssemblyCleanup methods in addition to the other methods, which basically all just write to the debug window that they are executing. What will be the output if you run all (both) these tests in debug mode?

As you may have guessed from the headline, the tricky part is ClassCleanup. Before I began to think about it, I just assumed that the test run would spin up TestClass1, execute the tests in that class and clean it up, then proceed with TestClass2 in the same manner.

Obviously, I was forgetting that unit tests are not ordered. Although they are executed sequentially, the order in which they are executed is not guaranteed. This is, in fact, a good thing, because it helps you remember that test cases should always be independent.

In any case, here's the result from my Output Window:

AssemblyInitialize
TestClass1: ClassInitialize
TestClass1: TestInitialize
TestClass1: MyTestCase1
TestClass1: TestCleanup
TestClass2: ClassInitialize
TestClass2: TestInitialize
TestClass2: MyTestCase2
TestClass2: TestCleanup
TestClass1: ClassCleanup
TestClass2: ClassCleanup
AssemblyCleanup

It's hopefully no surprise that AssemblyInitialize runs first, and AssemblyCleanup runs last. Notice, however, that although TestClass2's ClassInitialize execution is deferred until needed, this doesn't mean that TestClass1's ClassCleanup executes immediately after the last test case in the class! In fact, it waits until all test cases are executed, and the executes together with TestClass2's ClassCleanup.

This surprised me at first, but that was obviously only because I hadn't really thought it through: Since tests are, in principle, unordered, there's not guarantee that all tests in TestClass1 are executed in immediate succession. Theoretically, the execution engine may pick a test case from TestClass1, then one from TestClass2, then another from TestClass1, etc. Since that is the case, there's no guarantee that all tests from one test class have been executed before a new test class is initialized, and thusly, all ClassCleanup methods may as well be deferred until all test cases have been executed.

When this first surprised me, I had written some code in one class' ClassCleanup and expected it to run before another class' ClassInitialize. What I got instead was a strange interdependence bug, where all tests in both classes succeeded when executed together with other tests from their own class, but as soon as I ran all tests from both classes, the tests in the second class failed. The lesson here is that not only should you treat your test cases as independent, but the same goes for initialization and clean-up code.