Windows Phone Unit Tests in Visual Studio 2012 Update 2
January 31, 2013
Unit Testing is an important part of the application development life cycle for any platform. Visual Studio users no longer need to depend on 3rd party frameworks for unit tests on Windows Phone. The Windows Phone Unit Test framework delivered in CTP2 of Visual Studio Update 2 provides a consistent experience for developers to author, run and debug tests for Windows Phone 8 platform. This article walks you through the steps to start unit testing on Windows Phone.
Prerequisites
- Visual Studio 2012
- Windows Phone SDK 8.0
- Visual Studio 2012 Update 2 CTP 2
Note: You need a 64 bit Windows 8 physical machine (with Hyper-V enabled) to run unit tests on Windows Phone Emulator.
One Framework – Multiple Platforms
You can now write Unit Tests for Windows Phone in C# by creating a new “Windows Phone Unit Test App” project under Visual C#/Windows Phone. Tests are executed on the target device by installing the Unit Test app on the phone.
Figure 1: New Windows Phone Unit Test Template
Whether you are writing classic unit tests, Windows Store unit tests, or Windows Phone unit tests – Visual Studio automatically adds all the necessary references and you are ready to start writing tests using the familiar MS Test framework. You no longer need to learn multiple frameworks to author unit tests for different platforms.
Figure 2: Author tests using MSTest framework
Running Tests from Visual Studio
The Phone UT framework is designed as an adapter on top of the extensible Visual Studio 2012 unit testing platform (similar to other adapters like nUnit, xUnit). With this, the whole Visual Studio Test Driven Development experience is now extended to Windows Phone. Tests are discovered and displayed in the familiar Test Explorer window. Right from within the IDE, users can group/filter tests using traits, turn on continuous test runner to view test results after each build, and debug or run selected (or failed) set of tests.
|
Figure 3: Phone Unit Tests discovered |
Figure 4:Set Target Device |
Visual Studio uses the target device set on the Run palette to install the unit test app, run tests, and return results. Supported targets include Emulators (only available when Hyper-V is enabled) and Windows Phone 8 devices physically connected to the machine. Developers can author, run, and debug tests on any of the target devices without leaving the IDE. While unit tests are running on the device, a ‘Tests Running’ screen will be shown.
Figure 5: Unit Tests running on device
Running Tests from Command Line
As I mentioned earlier, since Phone UT is designed as an adapter to unit testing platform, you can use vstest.console.exe to execute unit tests from command line. You can also choose to export the results to Visual Studio Test Results File using /Logger:trx.
Figure 6: Run tests from command line
Figure 7: Test Results from TRX logger
By default Emulator WVGA 512MB is the target device to execute tests, use run settings to change target device (Sample run settings attached).
Figure 8: Run Settings to set target device
FAQs and Known Issues
- Phone Unit Test does not support data collectors either on the physical device or emulator.
- Can I write unit tests in native C++?
In CTP2 only C# is supported. VB# and C++ are not supported for now. - Which versions of Windows Phone is supported?
You can only author unit tests for apps that target Windows Phone 8. Windows Phone 7, 7.5, 7.8 are not supported. - Which Visual Studio SKUs do I need to enable this feature?
Windows Phone Unit Tests works on all editions – Express, Test Professional, Professional and Ultimate. - After installing all the prerequisites, I still do not see Windows Phone Unit Test template
Workaround: Install the mentioned prerequisites in the specified order, all on the system drive - Device was disconnected or interrupted while unit tests were in progress, what do I do?
Workaround: When a device is disconnected or interrupted (for example, by an incoming call or the camera button), the test run fails and vstest may crash. Manually uninstall all the unit test applications from the device, reconnect the device, and try running tests again. - I get an error message while debugging unit tests.
Workaround: Manually stop the debugging session at the end of a test run and start debugging selected tests again. - Phone Unit Tests that have a native dependency cannot be run from command line
- After Test Runs are complete, I sometimes still see Unit Test applications installed on device
Workaround: Manually uninstall all the unit test applications from the device. - Known Issue: Users who have a non English version of Windows Phone SDK Tools 8.0 will not be able to create new Windows Phone Unit Test projects.
- Known Issue: When a test fails, the stack trace does not contain line numbers indicating where the test failed, and users cannot navigate to code from stack trace information.
Any plans to bring this to WP7? FYI, this doesn't send a very positive message…
What's the point in creating and deploying test app to run unit tests? Usually tests cover some business logic or algorithms and are fine to run as any other test. Can you share some scenarios where this feature may be useful?
Thanks for bringing this much-needed feature to Visual Studio!
@andreycha: Theoretically you are right, but in reality just too many methods/code paths of a phone application depend in some way or another on namespaces that are only available on the phone/emulator. Except for the smallest algorithmic methods none of this code would be runnable and testable in VS directly.
Great post Vinay!
Congrats on getting your first one out!
@friederbluemle I see. So creating of test app is just a way to run that code in proper environment?
@Richard Szalay: Owing to architectural constraints, this remains a WP8 only feature.
@andreycha: Yes – ideal unit tests should mock all Phone APIs to be able to run tests in VS. In absence of that, the next best thing is to run tests on the device itself.
@Chuck: Thanks! 🙂
Trying this out… and couldn't get the test to run (after a long pause, the test runner stops and then leave the tests in "Not Run Tests" group). So I went to try the command line instead (what's up with it saying Windows Store apps instead of Phone, btw? :)), and it gives following error (am trying on device – I don't currently have Hyper-V to use emulators):
Starting test execution, please wait…
Warning: Using Isolation mode to run unit tests for Windows Store apps. Use /InI
solation switch to suppress this warning.
Warning: Diagnostic data adapters are not supported when executing unit tests fo
r Windows Store apps. Remove diagnostic data adapters settings from settings.
Error: Failed to initialize client proxy: could not connect to .
Error: The socket connection was aborted. This could be caused by an error proce
ssing your message or a receive timeout being exceeded by the remote host, or an
underlying network resource issue. Local socket timeout was '00:00:59.9990004'.
Hi Vinay!
Can one let the windows phone unit tests run in a Team Foundation Build Server (maybe inside of vstest.console.exe)?
Regards,
Alexander
@Vinay.Krishna Any chance you could elaborate on those "architectural constraints"? Since the tests run on the phone, I assume you're just hosting an HTTP service to communicate the results. I've done this for the WP7 (wp7ci on Codeplex) so I know it's possible, albeit without the VS integration.
@Wiyono: Since you are running tests on a processor without SLAT capability, the Target defaults to Device. Can you set Platform to ARM in Build -> Configuration Manager and give it a try? Also, while running tests from CLI, provide the arm xap and test settings with Target set to device. (I'm assuming your device is already unlocked with a developer license). And we are fixing up the names in messages.
@Alexander Soutchilin: You cannot run Phone Unit Tests as part of Team Build yet – we have this in our backlog. However when you execute tests from CLI, you can use /logger:TfsPublisher to publish results to Build Summary Page.
@Richard Szalay: More than the communication channel, it's the .NET CLR for Windows Phone 8 that made it possible for us to port our existing unit testing solution to WP8 platform.
Thanks Vinay, turns out I overlooked to set Platform to ARM (or use the ARM xap for CLI) instead, indeed. Nice one! (although perhaps, error experience could've been more helpful…) Now that it works on device I have found it to be more reliable also it seems (I have since tried it on emulators on another machine and it seems flaky that way in that sometimes not all tests are run and I have to manually select and run each of those, despite having selected Run All – note they are not skipped). But thanks, I have a better workflow now that this works!
Regarding the known issue : "Phone Unit Tests that have a native dependency cannot be run from command line". Will this issue be fixed when update 2 is released?
Does the unit test framework provide a way to collection code coverage numbers?
@RobN75- Yes, referencing native dependency has been addressed in upcoming release.
@Julius Ang- No, WP8 Unit Test framework doesn't support code coverage collection.
Are UITestMethodAttribute and async Assert.ThrowsException coming to this WP8 test framework too? Perhaps CTP3 (please)? 🙂
Awesome.
But please explain why I would run my unit tests on the phone? It looks like I would use this for integration tests, in which I would prefer to run as much of my code as possible on an environment as close to the final device as possible. In contrast, in a unit test I would mock out any dependencies that I cannot control and test smaller units. With this framework I cannot use Rhino Mocks and probably also not SpecFlow.
Please elaborate on the intended scope of use for this framework!
@Brian Elgaard – Yes – ideal unit tests should mock all Phone APIs to be able to run tests in VS. In absence of that, the next best thing is to run tests on the device itself. Please read earlier comments from andreycha and friederbluemle.
@Wiyono – Thanks for your excitement. Unfortunately both UITestMethod and async Assert.ThrowsException will not be available in Update 2. Please vote for this idea on visualstudio.uservoice.com to help us prioritize our future backlog.
Thanks. Voted up (with maximum count) :). Meanwhile, I just came across another issue now. Knowing that there is a limitation currently with native dependency (as you mentioned, although I actually found not just on command line but also from the test explorer), I have created AnyCPU configuration that is built without native dependencies. However I now found that the framework doesn't seem to like it? It just hangs waiting for something it seems – I could see the test app deployed fine to the emulator and the app opens briefly and closes, and the test still keep waiting indefinitely. Any idea?
This is kind of useless if I cannot use Fakes. Any plan to support Fakes ?
I guess implementing your own logger will be possible in the future versions? Howabout unlocking the device automatically so that I can have fully unattended test execution in CI?
I would like to get this used in CI but the two above features are a must before we can do it.
Hi Vinay. When will the limitation of "tests that have a native dependency cannot be run from command line" be addressed? Thanks.
What's the best way to write unit tests for old school asynchronous methods like the below using VS 2012 Update 2 and MsTest?
Something like:
var request = new MyOldRequest();
request.Completed += result => Assert.IsTrue(result);
request.Failed += error => Assert.Fail(error.ToString());
request.Execute();
Seems like the AnyCpu configuration does not work, only x86. Why?
Hi, Unit test for windows phone 8 is very useful for us. And I want to integrate with CI server (Jenkins), but it shows
"Could not start test run for unit tests for Windows Phone app: Unit tests for Windows Store apps cannot be run from a service or non interactive process. Please run unit tests from an interactive process.."
I can run unit test with wp8 device by using command vstest.console, but cannot run it through Jenkins.
Any ideas to run with Jenkins?
Thanks a lot. ^_^
Can unit test cases be run manually.
The unit test app is getting installed, then test are running and after that is getting uninstalled if we run unit tests from either IDE/command prompt.
If I am running manually, the app is getting stuck at showing "Tests running."
My requirement is that after the test cases have been run, a file will get generated in the isolated storage which I need. But in this case, as unit test app is getting uninstalled after the test cases have run, I am not getting any file.
Pajace, did you find a solution for your problem?
Having problem while trying to create test methods using IValueConverter. Also getting following error:
The type 'System.Windows.Data.IValueConverter' is defined in an assembly that is not referenced. You must add a reference to assembly 'System.Windows, Version=2.0.6.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e'
After adding System.Windows dll from framework tab I getting same error.
I've read this article and tried to get running. Windows Phone 8.0 SDK is installed, but I do not see any Test App Projects. Is it, because I have Windows Phone 8.0 SDK – DEU (German version)? What am I missing to get the templates. They don't appear in VS 2012 and 2013 either.