Evolving the Visual Studio Test Platform – Part 3: .NET Core, convergence, and cross-plat

[This is the 3rd post in the series on evolving the Visual Studio Test Platform. You can read the earlier posts here:
Evolving the Visual Studio Test Platform – Part 2,
Evolving the Visual Studio Test Platform – Part 1]

As .NET Core draws an ever-growing community of developers with existing assets and experiences, it is essential to support a consistent tools ecosystem. Thus, the “alpha” release of the MSBuild-based .NET Core Tools shipping with Visual Studio 2017 RC introduces support for the MSBuild build system and the .csproj project format – both familiar and key components of the .NET tools ecosystem. The need for a consistent testing experience follows naturally. Thus, the .NET Core testing experience/ecosystem is now converged with the Visual Studio Test Platform.

Concretely, this means the following:
(1) Evolved test platform: vstest is now fully cross-plat (Windows/Linux/Mac). dotnet test now leverages this evolved Visual Studio Test Platform.
(2) Converged adapters: Adapter writers can maintain a converged adapter that can serve Visual Studio, the vstest.console.exe CLI and the dotnet CLI.
(3) Converged user experience: the experience of using vstest.console.exe (the same options and capabilities) on the .NET Framework will be available on .NET Core as well.

Let’s see how …

Evolved test platform

The evolved vstest engine is presently named Microsoft.TestPlatform.V2 and is packaged as a vsix extension. Launch Tools | Extensions and Update… and you will notice that this is already bundled with Visual Studio.

5
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions is the folder where extensions are installed.

Converged Test Adapter for .NET Framework and .NET Core

When we first introduced MSTest V2 support there were 2 adapters – dotnet-test-mstest to target .NET Core, and MSTest.TestAdapter to target the .NET Framework. The dotnet-test-mstest adapter understands the project.json based project format. With .NET Core now supporting .csproj, that adapter is no longer required! Instead, the MSTest.TestAdapter can be used!

Launch Visual Studio 2017 RC, and create a Unit Test Project (.NET Core). This creates an MSTest V2 based solution targeting .NET Core. Now, look at the references.

1

The solution is self-contained – it refers the vstest engine, the MSTest V2 test framework, and the MSTest.TestAdapter to discover/run such tests using the engine. And it is targeting .NET Core.

The same approach can be used by other test frameworks as well. For e.g., xUnit.net too has 2 adapters – dotnet-test-xunit to target .NET Core, and xunit.runner.visualstudio to target the .NET Framework, but with .NET Core now supporting .csproj, the dotnet-test-xunit is no longer required either. Instead, the xunit.runner.visualstudio adapter can be used. Here is a .NET Core unit test solution using the xUnit.net test framework:

2

The solution is self-contained – it refers the vstest engine, the xUnit.net test framework, and the xunit.runner.visualstudio test adapter to discover/run such tests using the engine. This same adapter is used when targeting both the .NET Framework and .NET Core.

[Note: the evolved test platform has the notion of a translation layer that enables interfacing with IDEs to perform common tasks like discover tests, run tests, get the results to display, etc. This will be released soon on NuGet. In the meantime, we worked closely with the xUnit.net team to implement this support using early bits. We look forward to working with the community to extend this support.]

Let’s continue with this solution in our brief tour of the functionality.

Testing using the Visual Studio IDE

The below figure illustrates testing from within the Visual Studio IDE.

34new

This is same experience provided in the case of tests targeting the.NET Framework.

Testing using vstest.console.exe

C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions is the folder where extensions are installed, remember? Locate vstest.console.exe under this folder.
Navigate to the folder containing the xUnit.net test project, and use vstest.console.exe to run the tests as shown:6newNotice that vstest.console.exe takes the DLL(s) as input – just like it does in the case of tests targeting the .NET Framework. Go ahead, try this out and see what other switches can be used!

Testing using the .NET Core CLI

You can try these steps with latest dotnet-cli builds, since a critical bug fix couldn’t make it to RC release.
Install the latest .NET Code SDK Preview build from here:  https://github.com/dotnet/core/blob/master/release-notes/preview3-download.md. I downloaded it to the folder D:\temp\dotnet on my machine.
Use dotnet test to run the tests as shown:7new
The command automatically picks up the .csproj file(s) and discovers and runs the appropriate tests using the referenced xunit.runner.visualstudio adapter.

Testing using VSTS

To run the tests using VSTS, I setup a local build agent with Visual Studio 2017, and these same .NET Core SDK bits.
Here is the build definition:8new

Queue up a build and you should see the report.9new

Summary

The .NET Core testing experience/ecosystem is now converged with the Visual Studio Test Platform. This is a huge benefit to everyone – same adapters, common invocation model, much broader feature set than CLI). As mentioned earlier, we will release the translation layer to NuGet soon, and look forward to working with you to get more adapters interfacing with the test platform.

Stay tuned!

3