Announcing MSTest V2 Framework support for .NET Core 1.0 RTM
September 1, 2016
MSTest V2 is now supported on .NET Core 1.0 RTM. API changes in .NET Core 1.0 leading up to RTM (see #7754), were blocking users of the MSTest V2 packages we had released for .NET Core 1.0 RC2. Not any more (see #10713), we are pleased to note.
This continues to be a preview release, and comes with following changes:
- dotnet-test-mstest is built against .NET Core 1.0 RTM.
- Deprecated support for netstandardapp1.5 moniker as announced here.
- Fixes for bugs reported by you:
- “the test suite returns an exit code of zero even when there are failing tests” (from a comment in our earlier post)
- “AssemblyCleanup throwing Exception in ASP.NET Core Project Targeting .NET 4.5.1” (from a comment on the forum)
Now let’s go through the same steps as enumerated in our earlier post.
Installing the SDK
Install the Visual Studio official MSI installer from https://www.microsoft.com/net/core.
Creating a class library project
Create a .NET Core Class Library application. Open Visual Studio, and choose File | New | Project:
Adding references for MSTest
From nuget.org, install the MSTest.TestFramework package (shown below).
Now, install the runner – look for the updated dotnet-test-mstest package, and install it:
Open the project.json
file in the solution. You will already see the packages you just installed mentioned under “dependencies”. Add the “testRunner” property and set that to “mstest”. To make it easier, just replace the content of the project.json
file with the following (and note the differences in version numbers from our earlier post):
{
"version": "1.0.0-*",
"testRunner": "mstest",
"dependencies": {
"dotnet-test-mstest": "1.1.1-preview",
"MSTest.TestFramework": "1.0.1-preview"
},
"frameworks": {
"netcoreapp1.0": {
"imports": [
"dnxcore50",
"portable-net45+win8"
],
"dependencies": {
"Microsoft.NETCore.App": {
"version": "1.0.0",
"type": "platform"
}
}
}
}
}
Writing the tests
Visual Studio would have automatically created a file named Class1.cs
. Open that file and replace its content with the following:
using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace SampleNetCoreUnitTests
{
[TestClass]
public class TestClass
{
[TestMethod]
public void TestMethodPassing()
{
Assert.IsTrue(true);
}
[TestMethod]
public void TestMethodFailing()
{
Assert.IsTrue(false);
}
}
}
Testing using the Visual Studio IDE
Open the Test Explorer window (Test | Windows | Test Explorer in Visual Studio).
Build the solution, and you should see the tests as follows:
Click on “Run All” to run the tests.
Testing using the .NET CLI
Open a command prompt and navigate to the folder containing the solution, and type dotnet test
to execute the tests:Use the
dotnet test --list
command to discover tests.
Use the
dotnet test --test
to execute specific tests.
Testing using vstest.console.exe
The venerable vstest.console.exe may be used as well to run tests. Open a Developer Command Prompt, and just point vstest to the project.json file using the built-in Vsix based adapter to execute the tests:
vstest.console.exe project.json /UseVsixExtensions:true
Use
vstest.console.exe project.json /UseVsixExtensions:true /logger:trx
to generate reports too.
Testing using VSTS
To execute the tests in VSTS CI, check in the code and create a build definition with the following steps:
- a Command Line step that invokes the
dotnet restore
(i.e. “Tool” set todotnet
and “Arguments” set torestore
) - a Visual Studio Build step with the default settings
- a Visual Studio Test step, with “TestAssembly” set to
**\project.json
, and “Other console options” set to/UseVsixExtensions:true /logger:trx
Queue up the build, and you should see the following:
Build completed, tests run (with one of the tests failing as expected), and test results published!
Feedback
Thank you for the feedback you have given us till now. Not all the bugs/issues you reported may be fixed yet, but our endeavor is to address them in subsequent drops. In the meanwhile, please continue to send us your feedback as comments on this post, or using Visual Studio’s “Report a Problem”/”Provide a Suggestion” features, or using connect, or twitter.
Looking forward to hearing from you.
Thanks for all of the team’s hard work on getting this up to speed!
Couple questions:
1) Could we get better naming on the packages? dotnet-test-mstest and MsTest.TestAdapter are really a terrible for discoverability. Maybe MSTest.TestFramework -> MSTest.Core, dotnet-test-mstest -> MSTest.Adapters.Console, and MSTest.TestAdapter -> MSTest.Adapters.UWP? Then if you have the package dependencies set up properly (which right now they are not) you can just tell people to install the write Adapter, and the MSTest.Core package will come in automatically.
2) When are proper VS Project Templates going to be available for the new framework? Right now it’s very difficult to set up unit testing for UWP class libraries when you don’t want to run the tests in the context of a UWP app.
Thanks!
Robert McLaws
VS & Developer Tech MVP
CEO, AdvancedREI.com
Robert,
(1) We are heading towards a converged framework, adapter tuple. Please see here for context: https://blogs.msdn.microsoft.com/visualstudioalm/2016/06/17/taking-the-mstest-framework-forward-with-mstest-v2/. Having said that, we will keep a close eye on naming to ensure it does not get confusing.
(2) Starting with Visual Studio “15” Preview 4 (https://blogs.msdn.microsoft.com/visualstudio/2016/08/22/visual-studio-15-preview-4/), the Unit Test Project Template defaults to the new framework. I will write on that soon.
Yes, it needs better naming. All the three: MSTest.TestFramework, MSTest.TestAdapter and dotnet-test-mstest. Also, it would be good if all of them are combined open sourced.
The Wi-Fi just iPad Air 2 is available in 16GB, 64GB and 128GB versions priced at £399, £479 and £559, respectively.
Great, I’m looking forward to use MSTest in .NET Core.
Can I use data-driven test in dotnet-test-mstest?
For example, we can use CSV data file to repeat the TestMethod for each data row.
Can I use this? If not, is there any plan?
[TestMethod()]
[DataSource(
“Microsoft.VisualStudio.TestTools.DataSource.CSV”,
“|DataDirectory|\TestData.csv”, TestData#csv”,
DataAccessMethod.Sequential)]
[DeploymentItem(@”TestData.csv”)]
public void VariousInputPatternsErrorTest() { … }
You could use something like that
[DataTestMethod]
[DataRow(1, 1)]
[DataRow(1, 2)]
public void TestHolidays2017(int month, int day)
{
…..
}
Is it possible to run the tests with the command line tool on linux or mac?
Hi Pratap. It’s great to see that you have released a new preview version of MSTest. I would love to see MSTest open sourced so that the community can add additional features and more importantly fix bugs. Any idea if MSTest is going to be open sourced?
https://visualstudio.uservoice.com/forums/121579-visual-studio-2015/suggestions/13399980-make-ms-test-framework-opensouce
Since you guys are making so many improvements anyway, while you are at it can you eliminate non-problem exceptions when debugging tests with “Just My Code” turned off?
Btw the “connect” link just goes to the connect home page and not the project this is in. I imagine it would be one of the listings of Visual Studio and .NET Framework, but which one? Searching for “mstest” returns nothing :(.
I thought .net Core was supporting xUnit testing platform.
Great news!
Do you know when will be code coverage measurement working? I’m using xUnit and when I click Test/Analyze Code Coverage/All Tests, all my tests fail with error:
Test Outcome: Failed
Test Duration: 0:00:00
Result Message: Could not load file or assembly ‘Microsoft.VisualStudio.CodeCoverage.Shim, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’. Cannot find specified file.
Or perhaps it is working now with MSTest v2?
Hi,
Why not write something on how MSTest compares to NUnit/xUnit, what advantages it offers, etc?
Thanks!
+1 on open sourcing the MSTest Framework, especially the full .net framework version not just .net core. While you are at it, please open source the codedUI tools and UI Testing frameworks!
Does that mean the CodeCoverage will be updated as well? Last I had checked they have not had any development since 02/2016: https://www.nuget.org/packages/Microsoft.CodeCoverage
Here the docs only mention xUnit (to @Israel’s comment too), is the MsTest II now de facto going forward?: https://docs.microsoft.com/en-us/dotnet/articles/core/testing/unit-testing-with-dotnet-test?hubRefSrc=email&utm_source=lfemail&utm_medium=email&utm_campaign=lfnotification
Thank you all for the efforts on bring this to the community (not just the open one) 🙂
There is a new release of MSTest V2 on NuGet. See here: https://twitter.com/pvlakshm/status/778888081298423808
@nakama,
CSV as a datasource is not supported. Yet.
@DaveAbramson,
Not yet. Not too far away either.
@Adam, Johnathan,
CodeCoverage is coming soon.
@Ricardo,
These frameworks occupy their own place/niche, and offer their own advantages. As I mention here (https://blogs.msdn.microsoft.com/visualstudioalm/2016/06/17/taking-the-mstest-framework-forward-with-mstest-v2/), As a test platform we take a pluralistic approach, leaving the choice of test framework to the customer.
@Andrew, @Nick,
We are taking a hard look at this. Kindly consider voting here: https://visualstudio.uservoice.com/forums/121579-visual-studio-2015/suggestions/13399980-make-ms-test-framework-opensouce.
Thank you for the great work @Pratap & team. ANy plans to support .orderedtest files in these projects? I have a need to control the order of 20+ tests in one of my projects that I am migrating to .Net Core. Without it its a No Go for me.
I think also missing is the AsyncAssert to properly handle the tests where Assert.ThrowsException is calling an async function. Looking forward to another version with these issues fixed.
I hit a wall on this for a bit myself. I tried making the lambda argument an async one, but sometimes the tests would pass while most of the time, those tests would not run at all (while my synchronized method tests all worked as expected). I ended up adding GetAwaiter().GetResult() to my lambda, and it will let you test those methods.
@mvadu,
ordered tests are not yet supported for MSTest V2 based tests.
I have added this as a suggestion on uservoice here: https://visualstudio.uservoice.com/forums/121579-visual-studio-2015/suggestions/16417609-enable-ordered-test-support-for-mstest-v2-framewor. Please consider voting for it and spreading the word for others to vote for, and comment upon.
We will look into the support for AsyncAssert.
Thank you.
Can MSTest be debugged from VS Code, yet?
Hi
I get following error with “MSTest.TestFramework”: “1.0.5-preview” and “dotnet-test-mstest”: “1.1.1-preview”
Exception thrown while executing test. If using extension of TestMethodAttribute then please
contact vendor. Error message: Could not load type ‘LogMessageHandler’ from assembly
‘Microsoft.VisualStudio.TestPlatform.TestFramework, Version=14.0.0.0, Culture=neutral,
PublicKeyToken=b03f5f7f11d50a3a’
To execute the tests in VSTS I have to change the build definition:
Step 1 and 2 are ok
Step 3 TestAssembly” set to **Testproject.json, and “Other console options” set to /UseVsixExtensions:true
I get a System.IO.IOException because dotnet-test command was executed on source project. That was the reason why I changed TestAssembly field.
/logger:trx is automatically added by build step
**Testproject.json is neccessary
DanH,
Not yet. Let me know if that is important/urgent for you.
@Michael,
if you are using “dotnet-test-mstest 1.1.1-preview”, then please use “MSTest.TestFramework 1.0.4-preview”. We have not yet updated dotnet-test-mstest to work with the latest framework.
Fantastic work! and +1 for debugging
@Michael, Matt: This issue has hence been fixed with the release of dotnet-test-mstest 1.1.2-preview (http://www.nuget.org/packages/dotnet-test-mstest/1.1.2-preview) and MSTest.TestFramework 1.0.6-preview(http://www.nuget.org/packages/MSTest.TestFramework/1.0.6-preview)
Many thanks! will check it out today
Upgrading .net core to 1.1 mstest doesn’t work anymore.
I got this error:
Can not find runtime target for framework ‘.NETCoreApp,Version=v1.1’ compatible with one of the target runtimes: ‘win10-x64, win81-x64, win8-x64, win7-x64’. Possible causes:
“dotnet-test-mstest”: “netstandard1.6”: dont work????
The test projects targeting ASP.NET core are console applications and hence target netcoreapp1.0. Since netcoreapp1.0 has a compatible API set with netstandard1.6 you can add tests without an issue for a library you might have that targets netstandard1.6. Please use the template for project.json from the blog above.
-Abhitej.
@Gianluca: Can you try it out with a project.json similar to the below:
{
“version”: “1.0.0-*”,
“dependencies”: {
“MSTest.TestFramework”: “1.0.7-preview”,
“dotnet-test-mstest”: “1.1.2-preview”
},
“testRunner”: “mstest”,
“frameworks”: {
“netcoreapp1.1”: {
“dependencies”: {
“Microsoft.NETCore.App”: {
“type”: “platform”,
“version”: “1.1.0”
}
},
“imports”: [
“dotnet5.4”,
“portable-net451+win8”
]
}
}
}
and this global.json config:
{
“projects”: [ “src”, “test” ],
“sdk”: {
“version”: “1.0.0-preview2-1-003177”
}
}
Do let us know if this doesn’t work.
-Abhitej.
Are parameters from .runsettings file supported? I tried to access context.Properties[“Endpoint”] and got null.
unfortunately, this is not supported on VS 2015. This however should work with VS 2017.