Running Unit Tests for Windows Metro style apps from Command Line

Visual Studio 11 Beta was announced sometime back along with Win 8 consumer preview in which Unit testing has been redefined and support for testing Windows Metro style apps is added.

Tests for Metro style apps can be run from Visual Studio, Command Line and Team Build. Steps to create and run unit test from Visual Studio and Team Build are described in Walkthrough: Creating and Running Unit Tests for Metro style Applications and Running Windows 8 Metro style tests in TeamBuild respectively.

This post explains how to run Unit Test for Metro style apps from Command Line tool (vstest.console.exe)


Installing Unit Testing Command Line tool

Unit testing command line tool get installed with Visual Studio 11 Beta. Command Line test runner, vstest.console.exe, is installed at $(VSInstallDir)/Common7/IDE/CommonExtensions/Microsoft/TestWindow. It’s also get installed with Remote Debugger SKU.


Creating App Package for Unit Test Library

Create a Unit Test Library Project for Managed (C#/VB) or Native (C++) in Visual Studio. Steps for creating project are same as in Walkthrough: Creating and Running Unit Tests for Metro style Applications.

After tests are authored and ready to run from command line create an app package (.appx file) as follows:

  1. Right click on the project in Solution Explorer.
  2. Go to Store -> Create App Package…
  3. Select “Create a package to use locally only” and click Next
  4. Set desired Package location and Build configuration
  5. Click Create
  6. Click OK

Inside a folder in “Package location” specified in above steps following files can be found:

  1. <filename>.appx -> App package file for test project. It contains test binary and all Unit Testing platform binaries.
  2. <filename>.cer -> Certificate file corresponding to .pfx file used to sign package. It needs to be added to trusted certificates to install package.
  3. <filename>.appxsym -> Package containing public symbols of app package. (Not required for our scenario)
  4. <filename>.bat -> Batch file created with package to install package. (Not required for our scenario)

If you want to execute test on some other system (like ARM machine with Remote Debugger SKU) then copy .appx and .cer files on to that machine.


Executing Unit Tests for Metro style apps from command line

Install Dev License:

Install developer license on the box where test needs to be executed. This is needed to run unit tests for Metro style apps. This is onetime task per machine (may require renewing license, if expired).


Installing the certificate:

On machine where want to run test from command line, install certificate. This is a one time operation per certificate and can be done as follows:

a. From an elevated command prompt execute command “Certutil -addstore root <filename>.cer”


Executing tests:

Test for Metro style app can be executed from command line as follows:

“<TestToolPath>\vstest.console.exe” “<AppxPath>\<filename>.appx” /InIsolation 

<TestToolPath> is location where vstest.console.exe is installed (VS installs it at $(VSInstallDir)/Common7/IDE/CommonExtensions/Microsoft/TestWindow).

<AppxPath> is where App Package for test is present.

/InIsolation is required as tests for Metro style apps can’t be run in inproc mode. If option is not given then it will generate a warning.


Command line test runner (vstest.console.exe) will install app package, execute tests and uninstall package.


More options:

/Logger:trx option can be used to generate trx file which can be opened in VS to analyze failures.

/TestCaseFilter can be used to selectively run tests. More info can be found at TestCaseFilter in VS11 Unit Testing

/Tests & /ListTests options are not valid for Metro style app


Common Errors and Resolutions:

a. Don’t give dll file directly for test execution. Directly giving dll file will not execute test in appcontainer mode (in which Metro style app executes). Create app package as described above and use it for test execution in appcontainer mode.

b. Test project has its own manifest file and it is not derived from references. Set proper capabilities required to run tests in manifest file of test project.

c. To execute test as 64 bits on a 64 bits machine, use /platform:x64 option. By default, tests are executed as 32 bits on 64s bit machine.

d. If you are taking dependency on some SDK (like C++ test project takes dependency on VSLib SDK) then required dependency packages are copied in “Dependencies” folder with app package by package builder. If copying app package to another location or machine then copy “Dependencies” folder along with package.


If you’ve any feedback or issue then please do send us on Connect or in the Forums.

Comments (8)

  1. Anon says:

    Hey There,

    In the steps above, you describe how to create the App package for the Unit Testing Library as follows:

       Right click on the project in Solution Explorer.

       Go to Store -> Create App Package…

       Select "Create a package to use locally only" and click Next

       Set desired Package location and Build configuration

       Click Create

       Click OK

    I'm trying to run unit tests as part of a continuous build process, so I need a way to automate this. It is possible to do this from the command line?

  2. Hi,

    If you are using VS11 team build then you can follow steps in…/running-win8-metro-style-tests-in-teambuild.aspx .

    Give commandline argument /t:Publish to msbuild to generate app package from commandline. Use other parameters of msbuild to choose configuration.

    msbuild <solution.sln> /t:Publish


  3. hcmehta says:

    really worth content…vikram

  4. icu says:

    This is not working for me on VS2012 Update 2. I have a test project that runs perfectly fine in test explorer and it fails miserably from the command line.

    My command line looks like this:

    vstest.console.exe c:somepathmytestpackage.appx /Logger:Console /UseVsixExtensions:true /InIsolation /Platform:X86

    All I get is the generic-and-useless message:

    Error: The active Test Run was aborted because the execution process exited unex

    pectedly. Check the execution process logs for more information. If the logs are

    not enabled, then enable the logs and try again.

    I enabled logs by setting the registry values mentioned here:…/bb787181(VS.85).aspx

    And I got that url from a less-useless error message that I got before.

    Bottom line, I could never get this to work.

  5. Yamini L says:


    Can you please tell me if should explicitly add all the native test platform binaries, i mean the vstest.container, cppunittestframework dlls while attempting to create an appx package through command line ?

    also, do i need to copy all the binaries of my own app project to the unit test directory ? I'm not able to get it work if i don't do all this.

    CAn you please help on that ?

    Thanks !!

  6. Ajay says:

    Can you explain how can I read command line parameters in unit test library ? I don;t see any main function where I get command line parameters or Is there any other technique to read cmd line parameters.

    As well, Is there any way to pass custom/user specific command line parameters to unit test just like below

    unittestapp /myparameter:"somevalue"

  7. Vasanth says:


    We have VS 2013 and TFS 2010 so I am trying to run Unit tests via command in tfs build definition using Invoke process activity.  All the time it fails with Unauthorized exception, any idea how to fix it?

    Unhandled Exception: System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))

    at Microsoft.VisualStudio.TestPlatform.Utilities.IPackageDebugSettings.EnableDebugMode(String packageFullName, String debuggerCommandLine, String environment)

    at Microsoft.VisualStudio.TestPlatform.Utilities.AppContainerTestExecutorLauncher.LaunchTestExecutor(FrameworkVersion frameworkVersion, Architecture architecture, IDictionary`2 environmentVariables)

    at Microsoft.VisualStudio.TestPlatform.Client.TestExecutorServiceClient.Initialize_NoLock(Architecture targetPlatform, FrameworkVersion targetFramework)

    at Microsoft.VisualStudio.TestPlatform.Client.TestExecutorServiceClient.EnsureInitialized(IDictionary`2 environmentVariables, TestExecutionContext context)

    at Microsoft.VisualStudio.TestPlatform.Client.TestExecutorServiceClient.<>c__DisplayClass9.<StartTestRun>b__8()

    at Microsoft.VisualStudio.TestPlatform.Client.TestExecutorServiceClient.InvokeTestExecutorServiceAction(Action action)

    at Microsoft.VisualStudio.TestPlatform.Client.TestExecutorServiceClient.StartTestRun(IEnumerable`1 sources, TestExecutionContext executionContext, String runSettings, IDictionary`2 environmentVariables, String dataCollectionServiceUri, ITestCaseLog testCaseLog, ITestRunLog testRunLog, IFrameworkHandleInternal frameworkHandle)

    at Microsoft.VisualStudio.TestPlatform.Client.TestRunRequest.ExecuteAsync()

    at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.RunTestsArgumentProcessor.RunTests(IEnumerable`1 sources, String packageFullName, String deviceInfo, Boolean isPhoneAppx)

    at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.RunTestsArgumentProcessor.ExecuteAppContainerSources(List`1 appContainerSources, String deviceInfo, Boolean isPhoneAppx)

    at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.RunTestsArgumentProcessor.Execute()

    at Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.Execute(String[] args)

    at RockSteady.Program.Main(String[] args)