That Pesky MSTest Execution Ordering..


I have come upon several groups who are puzzled by the intricacies of execution of the MSTest Framework.  I admit – it is quite confusing, so I hope that these postings will help to clean some of the confusion.


I think that most confusion comes from some user’s expectation of MSTest to execute like the Nunit framework.  They execute differently since Nunit instantiates a test class only once when executing all the tests contained in it, whereas MSTest instantiates each test method’s class separately during the execution process, with each instantiation occurring on a separate thread. This design affects 3 specific things which often confuse users of MSTest:


1.      ClassInitialize and ClassCleanup: Since ClassInitialize and ClassCleanUp are static, they are only executed once even though several instances of a test class can be created by MSTest.  ClassInitialize executes in the instance of the test class corresponding to the first test method in the test class. Similarly, MSTest executes ClassCleanUp in the instance of the test class corresponding to the last test method in the test class.


2.      Execution Interleaving: Since each instance of the test class is instantiated separately on a different thread, there are no guarantees regarding the order of execution of unit tests in a single class, or across classes. The execution of tests may be interleaved across classes, and potentially even assemblies, depending on how you chose to execute your tests. The key thing here is – all tests could be executed in any order, it is totally undefined.


3.      TextContext Instances: TestContexts are different for each test method, with no sharing between test methods.


 


For example, if we have a Test Class:








    [TestClass]


    public class VSTSClass1


    {


      


        private TestContext testContextInstance;


 


        public TestContext TestContext


        {


            get


            {


                return testContextInstance;


            }


            set


            {


                testContextInstance = value;


            }


        }


 


 [ClassInitialize]


        public static void ClassSetup(TestContext a)


        {


            Console.WriteLine(“Class Setup”);


        }


 


        [TestInitialize]


        public void TestInit()


        {


            Console.WriteLine(“Test Init”);


 


        }


 


        [TestMethod]


        public void Test1()


        {


            Console.WriteLine(“Test1”);


        }


 


        [TestMethod]


        public void Test2()


        {


            Console.WriteLine(“Test2”);


 


        }


 


        [TestMethod]


        public void Test3()


        {


            Console.WriteLine(“Test3”);


        }


 


        [TestCleanup]


        public void TestCleanUp()


        {


            Console.WriteLine(“TestCleanUp”);


        }


 


        [ClassCleanup]


        public static void ClassCleanUp()


        {


            Console.WriteLine(“ClassCleanUp”);


        }


    }


(This consists of 3 Test Methods, ClassInitialize, ClassCleanup, TestInitialize, TestCleanUp and anexplicit declaration of TestContext)


The execution order would be as follows:


Test1 [Thread 1]: new TestContext -> ClassInitialize -> TestInitialize -> TestMethod1 ->  TestCleanUp


Test2 [Thread 2]: new TestContext -> TestInitialize -> TestMethod2 ->  TestCleanUp


Test3 [Thread 3]: new TestContext -> TestInitialize -> TestMethod2 ->  TestCleanUp -> ClassCleanUp


The output after running all the tests in the class would be:


Class Setup


Test Init


Test1


TestCleanUp


Test Init


Test2


TestCleanUp


Test Init


Test3


TestCleanUp


ClassCleanUp


 (A special thanks to Dominic Hopton for his edits of my hackery on this post :))

Comments (14)

  1. Naderi é PM (Program Manager) do VSTS Test Team e escrever bastante sobre testes unitários utilizando

  2. FredericALTorres says:

    Running more than one test at the same time is great.

    But are you planning a mode to define a special order of execution ? (Test dependency)

  3. jisotupa says:

    I extended the test a little bit to see how MSTest behaves with VS2005 and multiple test classes in different test projects. I also added code to print on what thread the methods get called.

    This is what I see when I run the tests from the Test View (please note the nice typo 🙂

    —————

    Class Setup 1 (on thread AdpaterExeMgrThread1)

    Test Init 1 (on thread AdpaterExeMgrThread1)

    Test 1.1 (on thread AdpaterExeMgrThread1)

    TestCleanUp 1 (on thread AdpaterExeMgrThread1)

    Test Init 1 (on thread AdpaterExeMgrThread1)

    Test 1.2 (on thread AdpaterExeMgrThread1)

    TestCleanUp 1 (on thread AdpaterExeMgrThread1)

    Test Init 1 (on thread AdpaterExeMgrThread1)

    Test 1.3 (on thread AdpaterExeMgrThread1)

    TestCleanUp 1 (on thread AdpaterExeMgrThread1)

    Class Setup 2 (on thread AdpaterExeMgrThread1)

    Test Init 2 (on thread AdpaterExeMgrThread1)

    Test 2.1 (on thread AdpaterExeMgrThread1)

    TestCleanUp 2 (on thread AdpaterExeMgrThread1)

    Test Init 2 (on thread AdpaterExeMgrThread1)

    Test 2.2 (on thread AdpaterExeMgrThread1)

    TestCleanUp 2 (on thread AdpaterExeMgrThread1)

    Test Init 2 (on thread AdpaterExeMgrThread1)

    Test 2.3 (on thread AdpaterExeMgrThread1)

    TestCleanUp 2 (on thread AdpaterExeMgrThread1)

    ClassCleanUp 1 (on thread TestCasemanager::ExecutionThread)

    ClassCleanUp 2 (on thread TestCasemanager::ExecutionThread)

    —————

    So all tests are run on the same thread and only the cleanup routines are called from a different thread. The class clean up methods are postponed until all test methods from all test classes have been run, which in my opinion makes the class initialize / clean up methods pretty useless unless your unit tests are very cleanly isolated.

  4. jisotupa says:

    Hmm, it seems that MsTest just does not name the threads – using the ManagedThreadId property we see something else:

    Class Setup 1 (on thread 17)

    Test Init 1 (on thread 17)

    Test1.1 (on thread 17)

    TestCleanUp 1 (on thread 17)

    Test Init 1 (on thread 19)

    Test1.2 (on thread 19)

    TestCleanUp 1 (on thread 19)

    Test Init 1 (on thread 20)

    Test1.3 (on thread 20)

    TestCleanUp 1 (on thread 20)

    Class Setup 2 (on thread 21)

    Test Init 2 (on thread 21)

    Test2.1 (on thread 21)

    TestCleanUp 2 (on thread 21)

    Test Init 2 (on thread 19)

    Test2.2 (on thread 19)

    TestCleanUp 2 (on thread 19)

    Test Init 2 (on thread 17)

    Test2.3 (on thread 17)

    TestCleanUp 2 (on thread 17)

    ClassCleanUp 1 (on thread 16)

    ClassCleanUp 2 (on thread 16)

  5. johanw says:

    For me this issue/design is making MS Unit tests pretty much useless and might cause me to think about starting to use nUnit again. The whole purpose of especially TestInitialize and TestCleanup is to run as pairs BEFORE the next test executes. If you have any database setup that you later change in your test it could cause a problem with the next test. Therefore it is imparitive that each test runs one at a time and always wraps the intitialize/cleanup pairs.

    The idea of having tests run at the same time on different threads is a dream that I don’t think can be unless you have a simple app that doesn’t have any advanced database setup. This is not the apps I at least tend to work on.

    Saying that you should switch to ordered tests isn’t a good idea either. Test driven development practices usually mandates that you have a unit test drive your entry point into the code you are developing/changing. Therefore you want to just go to the individual test you are currently working on and debug through it. With ordered tests you might have to go through a massive amount of tests before you even get there. Not a good idea.

  6. Насколько я помню, в классике модульного тестирования принято, что порядок выполнения модульных тестов

  7. That is totally a pain says:

    Sorry, but this is mad.

    I would like to define the tests in one specific order, so if one fails… there is no point in go on testing.

    It looks like the MS developers do not use their own products in real life scenarios.

    It is just disappointing.

  8. Nilzor says:

    So why don't you just use NUnit then, if you're dependent on execution order? Also I'm sure a lot of people disagrees in your statement "there's no point in go on testing" if one test fails.

  9. ac says:

    Your unit tests should never abort simply because one failed… That defeats the purpose of unit tests entirely.

  10. KMote says:

    Nearly 7 years later, this post is still useful. But it could be improved by adding the test class constructor:

    public VSTSClass1()

       {

           Debug.WriteLine("VSTSClass1 Constructor");

       }

    to demonstrate that the constructor is run (surprisingly) once per TestMethod.

  11. CptDD says:

    8 years later, still handy.

  12. Vitali Climenco says:

    There is also AssemblyInitializeAttribute. It decorates a static method that runs before any test in the assembly have run:

           [AssemblyInitialize()]

           public static void AssemblyInit(TestContext context)

           {

               // some code

           }

    A method decorated with AssemblyCleanupAttribute can be used to free resources claimed by the assembly, after all tests have run.

  13. Someone says:

    @That is totally a pain

    The scenario you described is not unit testing. Unit tests should work in any order and regardless of how are grouped. You should get the same result if you run a test individually, or as second in a test suite or as last in another test suite.

    You can do some of the other test types with MS Test, NUnit, XUnit and others but overally you'd be better off with test runners designed for your test types (i.e. acceptance tests, scenario based UI tests). Typically UI tests are where you'd abort test run in case a step fails, and if you translate this to MS Test/XUnit you'd rather have one test and more asserts.

    I.e. var session = service.Login(username, password)

    Assert.IsNotNull(session, "Authentication failed.");

    var data = service.GetSomeData(session);

    Assert.IsNotNull(data, "Failed to get data.");

  14. Someone says:

    @johanw

    It would be pointless to run TestInitialize and TestCleanup otherwise than are now. Set up your environment in TestInitialize, run the test, then after test clean up in TestCleanup. Why would you clean up before the test was run?

    A test what requires advanced database setup is not a unit test.