Extending Unit Tests in Visual Studio

It’s funny how you walk around with a question on your mind for a couple weeks and then one day it just suddenly feels like the ‘right time’ to go look for the answer.

Question: Does VS 2010 allow you to extend the (nice) integrated unit testing to run tests in custom ways?
Answer: Yes! Actually it was easy to find a guide with a sample [RunAsUnitTest] extension on the VS Team Test blog. [RunAs = ‘‘run as user’)]

The scheme is: define your own custom [TestClass] attribute, and VS will load tests using whatever execution engine the attribute provides. The article looks like it dates back to Beta versions of VS, and maybe something has changed but there were a few issues I had with the article.

Here are a few minor and major issues:

  1. You probably only actually need to add one assembly reference: Microsoft.VisualStudio.QualityTools.UnitTestFramework – which your unit tests will reference anyway

  2. It tells you to install your assembly to Visual Studio’s ‘PrivateAssemblies’ folder. However, this did not work for me. I ended up using ‘PublicAssemblies’ folder instead. Without this, I would get the message

    Failed to initialize the unit test extension 'urn:microsoft.adp.unittesting.stackoverflow.asdf': A unit test extension is not registered for the following attribute: System.Active.Test.StackOverflowTest.StackOverflowUnitTestAttribute.

    Maybe this was just because the UnitTestFramework assembly it depends is also in the PublicAssemblies folder, and we need dependent assemblies in the same folder?

  3. The HKEY_LOCAL_MACHINE registry key for registering custom type seemed to have no effect

  4. The HKEY_CURRENT_USER registry key for registering custom type seemed to be required

  5. When you get errors that the attribute type ‘T’ cannot be loaded from the Assembly ‘X’ you are on the right track: it did find your assembly. But you have either put the wrong type name in the registry key, or you are missing some dependency.

OK, on to practical applications. Why did I even think of customizing unit tests in the first place?

The original trigger of thought that unit tests which fail by encountering Stack Overflow (note: you should never have a passing test case involving catching StackOverflow, because StackOverflow means your process state just got corrupted) are not handled so well by Visual Studio’s default test host: the Windows error reporting dialog will appear. Obviously I wouldn’t care if all our test cases passed, but we are not that far along in our new project.

So: the idea is creating a test extension that can handle the stackoverflow exception more elegantly (by e.g. exiting, instead of crashing)?

Writing up the classes following the example was easy. But the experience of registering the type hurt, as you can guess from the above. I would have liked having my test extension be allowed as part of my test assembly, instead of registered. The article mentions that this is necessary because the test type is being loaded by Visual Studio itself, but that seems rather unnecessary considering that my extension doesn’t provide any custom Client-Side functionality. Perhaps in a future version there could be a ‘light’ run-time-only extension which VS doesn’t need to load?

Also unfortunately - in practice seems it is nearly impossible to catch StackOverflowException no matter what you do, but that one probably deserves its own blog entry or further research…