Authoring a new Visual studio unit test adapter

Visual studio 11 Beta unit test framework provides a way to plug-in third party test runners and run their tests right inside VS IDE. Here are the list of test adapters that are already available for download.

In this post, let's quickly go through the steps involved in writing a new test adapter.

A test adapter plug-in needs to have two basic components :

• TestDiscoverer - A component that gives you the list of test cases from your test source. The test source could be a .dll file or a .xml or anything. Given a test source a discoverer knows how to load and get the test case out of it.
• TestExecutor -A component that executes the tests. Given a list of tests, the executor executes them and gives back the results.

To design these components the VS Unit testing extensibility model provides you with two interfaces :

a. ITestDiscoverer

b. ITestExecutor

.

To implement these interfaces all you need to do is to create a class library and add a reference to "Microsoft.VisualStudio.TestPlatform.ObjectModel "assembly which would be found under "%Program Files%\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow" folder.

Let us walk through the implementation of a simple XML test adapter which would run tests in a given XML file. [Sample code attached with the post].

ITestDiscoverer defines just one method : "DiscoverTests". Given a list of test sources, this method would pull out the test cases from them.

If you take look at the parameters :

1. Sources --> Refers to the list of test sources that are passed to the test adapter from the client [VS or command line]. It would be a list of .xml files in our case.
2. Discoverycontext --> Refers to the discovery context/runsettings for the current test run. Discoverer shall pull out the tests in a specific way based on the current context.
3. Logger --> This is used to relay the warning and error messages to the registered loggers. Console Logger and TRXLogger are the built-in loggers that ships with Visual Studio 11. User shall register other custom loggers like FileStreamLogger or DBLogger as required.[ more details on writing custom loggers in future posts].
4. DiscoverySink --> Discovery sink is used to send back the test cases to the framework as and when they are being discovered. Also it is responsible for the discovery related events.

Discoverer Attributes:

The discoverer has two attributes associated with it:

FileExtension - Specifies one or more file extensions that the discoverer supports. It would be ".xml" in our case.

Executorurl - specifies the Url of the executor that is tied to the discoverer. The discover to executor mapping is 1:1.

Xml discoverer:

Lets quickly look at our sample XML discoverer. It scans the given XML source and identifies each "test case" based on XML Node attribute "Name".

It associates the executor (XMLExeuctor in our case) with each test case and sends the test back to the framework using discovery sink.

…..

…...

The ITestExecutor interface provides you with two simple overloaded methods to run tests :

1. RunTests ( given the sources)

This could be something which maps to the "RunAll" functionality. All the sources are given to the executor in one go and executor defines how to run them.

2. RunTests ( given a test collection)

This maps to "RunSelected" test functionality, where a bunch of test cases are passed to the executor.

The runContext object and frameworkHandle object are passed are parameters to RunTests method, which provides all the settings related to the current run.

XmlExecutor

Our XML executor just iterates through the given tests and sets the test results based the user specified "Outcome" value in the XML.

In this case RunTest(sources) overload  just calls the discoverer and calls RunTests(tests) overload. This could vary in other test adapters – say to have some optimizations for RunTests(sources).

Now that we are done with implementing the test adapter, we shall go ahead and register them. You shall register them using the steps mentioned here in Aseem's blog. Method 2 disucssed over there via VSIX is the generally recommended way to install extensions.

To quickly check if the test adapter has been registered properly, you can try "vstest.console.exe /listdiscoverers" from a Visual Studio command prompt. [vstest.cosnsole.exe is the new framework agnostic test runner.More details here on MSDN].

If you have installed your adapter as a VSIX make sure that you specify /useVsixExtensions:True switch in your command. The below image shows the XMLDiscoverer being listed along with the other built-in test discoverers.

Similar to /listdiscoverers, we have a /listexecutors switch as well.

Let’s quickly try running our tests with a sample xml test file as below :

Now the new test adapter is registered and we are able to run tests from command line. We should be able to run these tests from "Unit test explorer" window as well, if these tests are based on ".dll" or ".exe" extension.

Since our tests are based on ".xml", we need to implement one additional interface to make “Unit test explorer” aware of it. This is because, visual studio by default scans only ".dll" and ".exe" project items and passes them on to the registered discoverers.

Lets take a look at it in our next post.

MyXmlExtension.zip

• Anonymous
  July 08, 2013
  I've implemented both interfaces but no tests show up at Test Explorer and the breakpoint doesn't hit (attaching to vstest.discoveryengine.x86.exe). It is listed on vstest.console.exe /istdiscoverers /usevsixextensions:true
• Anonymous
  August 22, 2013
  For non .exe and .dll you need these interfaces as well ITestContainerDiscoverer and ITestContainerExamplegithub.com/.../TSTestExtension