vstest.console.exe - CommandLine Test Runner
As we know, the unit testing tools has been completely revamped in Visual Studio 2011. One such component is the new command line runner – vstest.console.exe
What is it ?
new command line test runner “vstest,console.exe” which would run tests written against any test framework – MSTest framework, Nunit, Xunit and so on.
Where can I find it ?
vstest.console.exe could be invoked from directly from “Visual Studio Command Prompt”. The executable would be found under “Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow” folder. Vstest.console.exe /? would give more details on the various switches provided by the test runner.
How is it different from Mstest.exe ?
With Visual Studio 2011, the tests written using visual studio unit test framework could be run using either vstest,console.exe or using mstest.exe. The behavior would be same with subtle differences (which is briefed in the below table).
vstest.console.exe has some additional options /platform /framework which would let the user to run their tests in a specific platform/framework to increase test coverage.
Mstest and Vstest:
Below is a brief summary on the various switches provided by Mstest.exe and the equivalent switches in vstest.console.exe
Mstest
Vstest.console
Example
/testcontainer
There is no switch needed in front of the input test assembly in vstest.console.exe unlike mstest.exe.
Example:
Mstest /testcontainer :abc.dll /testcontainer:xyz.dll
Vstest.console.exe abc.dll xyz.dll
/minpriority,
/maxpriority,
/category
/testcasefilter
Inv stest.console.exe /testcasefilter switch handles filtering based on all categories like priority , test category and so on.
Example:
Mstest /testcontainer:abc.dll /minpriority:1
Vstestabc.dll /testcasefilter:"Priority=1 | priority=2"
/testmetadata
/testlist
N/A
VSMDI file is deprecated in Visual Studio 2011 and there is no equivalent switch for this.
/testsettings
/runconfig
/settings
Example:
Vstest.console.exe accepts the run settings in two formats:
- .runsettings file which is the new settings format. .runsettings file can be used as configuration file for all test frameworks like Nunit, Vs unit, xunit and so on.
- .testsettings file which was present in Visual Studio 2010 and is specific to Visual studio unit test framework.
Example:
Mstest.exe /testcontainer:abc.dll /testsettings:nightly.testsettings
Vstest.console.exe abc.dll /settings:nightly.testsettings
/resultsfile
/settings
If the default location of results need to be overriden user need to pass this value using a runsettings file.
Example:
Mstest.exe /testcontainer:abc.dll /results:C:\Results.trx
Vstest.console.exe abc.dll /settings:output.runsettings where the context of the .runsettings file would be something like below :
<?xml version="1.0" encoding="UTF-8"?>
<RunSettings>
<RunConfiguration>
<ResultsDirectory>c:\</ResultsDirectory>
</RunConfiguration>
</RunSettings>
/test
/tests
Mstest.exe /testcontainer:abc.dll /test:API
Vstest.console.exe abc.dll /tests:API
/noisolation
In Visual Studio 2011, the command line runner runs tests in no isolation mode by default. Hence no switch is needed explictly.
/inisolation switch could be specified explicitly to force isolation if required.
Note : Though the default behavior is to run in process, if the input test assembly requires isolation (like needed a different .net platform or different architecture of the test runner, tests would be run in isolation)
Example:
Mstest.exe /testcontainer:abc.dll /noisolation
Vstest.console.exe abc.dll
/noresults
Vstest.console.exe doesn’t create the TRX file by default.
Results would be sent only to console. If TRX is needed, user has to enable it explicitly by specifying /logger:trx
Example:
Mstest.exe /testcontainer:abc.dll /noresults
Vstest.console.exe abc.dll
Mstest.exe /testcontainer:abc.dll
Vstest.console.exe abc.dll /logger:trx
/detail
Vstest.console.exe prints the stack trace and error message of each test to the console by default.
Hence there is no equivalent of /detail switch
Example:
Mstest.console.exe /testcontainer:abc.dll /detail:errormessage /detail:errorstacktrace
Vstest.console.exe abc.dll
/pulish and the related options
N/A
Vstest.console.exe doesn’t expose any functionality to publish results to TFS
/unique
N/A
There is no equivalent switch to unique. But the functionality could be achieved specifying the fully qualified name of the test.