Testing Windows 8 apps using Visual Studio 2012


As we continue to innovate on the operating system platform with Windows 8, we also enhanced the toolset available in Visual Studio to increase your productivity in testing your Windows 8 apps. Ensuring a top notch quality of your app in this new modern OS environment through thoughtful design and testing will help increase the success of your app. In a previous blog post on Testing Metro style apps in Windows 8, we highlighted some of the key areas you need to take into account for building a high quality Windows 8 app. In this post, we explore a few capabilities in Visual Studio 2012 that will make the testing and verification of your app easier.

How does my app look on Windows 8 devices?

While the success of any app hinges on the value it offers to its customers, an app that follows Windows 8 design guideline to provide a polished look and feel will delight the end users. To reduce your reliance on physical devices to verify the app’s user experience, Visual Studio 2012 provides a simulator so you don’t have to connect to a tablet device all the time during development. The simulator also reduces your need to have physical devices of every form factor at your disposal for testing. Your testing benefits in these areas where the simulator enables quick and convenient verification of your apps’ look and feel with just a few mouse clicks:

Visual Studio simulator displaying the Windows 8 start screenFigure 1. Visual Studio simulator displaying the Windows 8 start screen.

Windows 8 apps run on devices of multiple form factors. To help you test your app’s behavior in different screen sizes and resolutions, Visual Studio simulator provides 7 most common combinations of those two factors that cover from the minimum supported resolution to a 27” desktop monitor. Figure 1 illustrates an example of Windows 8 start screen shown in the simulator at the default resolution of 1366×768 and screen size of 10.6”. One of the first things to verify is that at all times the layout of your app is not clipped or misaligned vertically or horizontally at lower resolutions. On the other hand, you don’t want the app to leave a large amount of blank space at higher resolutions either. If your app’s layout extends beyond the screen, the scroll bar should show up appropriately if a user is interacting with the app using a mouse. These are examples of the type of verification you can easily accomplish by changing the simulated resolutions on the fly in the simulator and visually examining the behavior of the app.

If your app uses a fixed layout, you need to test whether it scales correctly to fill the screens of large sizes and make sure all the visual elements also scale proportionally. If your app has an adaptive layout, the simulator allows you to quickly verify that the app’s visual elements are dynamically rearranged as you intended when screen size or resolution changes.

The simulator can be rotated or flipped to see how your app behaves when a user changes the orientation of the screen. This comes in handy if you want to test such an app to see if it is correctly optimized for the portrait mode and the landscape mode. The simulator doesn’t eliminate the need to fully verify the app on physical devices, because it doesn’t simulate all the accelerometer sensitive capabilities such as tilt, but it will save you a lot of time in exposing potential problems in your app’s response to the changing of the orientation of the displaying device.

How does my app respond to touch on Windows 8 devices?

The simulator allows you to interact with your app using mouse or using simulated touch on your development computer. The latter is particularly handy if you don’t have a touch-capable device available for testing. Swipe, pinch to zoom and rotation gestures are all supported.

You typically launch your app in the simulator directly from within the Visual Studio through the debugging action. After the app is launched, the simulator stays running, which allows you to also exercise the app outside of the debugging context. For example, you can watch the activation behavior when a user taps your app’s tile or the termination behavior when a user swipes from the top to the bottom of the screen. You can take advantage of the touch gestures that the simulator provides to perform these actions and all the touch related testing and verification. Testing Metro style apps in Windows 8 has a detailed list of testing considerations related to touch. Refer to the Touch section in that blog post. If you want to learn more about the Visual Studio Simulator such as how to use the touch emulation, take a look at Running Windows Metro style apps in the simulator.

Unit testing your Windows 8 apps

The ability to unit test your code is an important aspect of any development cycle. Unit test support has been shipping in Visual Studio for several releases. In Visual Studio 2012, we have streamlined the user experience with a new unit test explorer and created an extensible test runner that is independent of the underlying unit test framework. Now you can plug in your favorite unit test framework such as NUnit, XUnit or MbUnit, or you can continue to use MSTest.

New Project dialog in Visual Studio 2012  Figure 2. New Project dialog in Visual Studio 2012 shows the template for the
Unit Test Library project for the apps written in C#.

For the Windows 8 apps written in C#, VB.net or C++, Visual Studio now provides a unit test library project that follows the same workflow you use for developing a desktop app (see Figure 2). You will enjoy the improvements and many of the same features you are familiar with. Under the hood, the unit test support takes care of running your tests in the app container model and abstracts all the complexity away. The unit test project has its own appxmanifest file. This is helpful because in many cases such as when you are testing the code that handles location service or internet access, for the tests to run, you need to edit the manifest file to specify the capabilities that match your app’s requirement, as illustrated in Figure 3. Alternatively, if the part of the code path under testing doesn’t need a capability, you can group such test cases in a separate unit test project and deliberately exclude a capability from it to improve the modularization of your code.

package manifestFigure 3. Package.appxmanifest file of a unit test project opened in the Manifest Designer. 

In keeping with our commitment to quality, we made the improvements and new functionalities in unit testing available in all the SKUs of Visual Studio 2012, including Visual Studio 2012 Express for Windows 8. For more details or to learn how to effectively unit test your Windows 8 app, see Creating and running unit tests on a Windows Metro style app and this unit testing talk from the //BUILD conference.

Exploratory testing your Windows 8 apps

Windows 8 apps are designed to excite consumers and businesses alike. If you work on an app in an enterprise environment, and are used to a more structured development and testing process, Visual Studio 2012 provides a manual testing tool, Microsoft Test Manager, which in combination with Remote Tools for Visual Studio, allows you to conduct exploratory testing and structured manual testing of your app’s functionality on all types of Windows 8 devices. Microsoft Test Manager is part of the Visual Studio Premium and Ultimate SKU and the Test Professional SKU. In addition to the seamless integration with TFS for easy bug filing and test case management, the manual testing tool in Visual Studio 2012 simplifies testing of your app on a remote device. During the testing process, the tool can capture screen shots, action logs and event logs from the remote device when you discover a bug in your app. It provides a two-way communication between your main workstation and the connected device that guides you through the steps of deployment of your app packages to the targeted testing device, acquisition of developer license on the device, and the execution of test steps.

With TFS integration, the manual testing tool enables a continuous workflow in a team environment between testing and development of your app. For example, the info captured during exploration helps you reproduce the bug and improves the understanding of its nature. When a new app package becomes available with bug fixes or feature enhancement, it can be easily picked up and deployed to the same Windows 8 testing device for further verification from within the manual testing tool.

For more info, see Visual Studio 11 Beta: Manual Testing of Windows Metro Style Apps. Manual Testing Of Windows 8 Metro Style Applications provides a detailed walkthrough.

Conclusion

Windows 8 is an exciting platform for developers and users. We hope that this blog is a useful introduction to the tools shipped in Visual Studio 2012 that you can start using right away to test the Windows 8 app you are working on. We look forward to hearing your feedback to see how we can continue to make that part of your life easier. Don’t forget to leave comments on this post to share with us and the community tips, ideas, and personal experience on how you have been testing your apps.

–Rui Sun, Principal Test Manager, Visual Studio

–Andre Hamilton, SDET, Visual Studio

Special thanks to Ashwin Needamangala, Mete Goktepe, Jake Sabulsky, Jason Olson, Art Becker, Raul Gonzalez Tovar, Peter Provost, Mathew Aniyan, and Anutthara Bharadwaj for their contributions to this post.

Comments (25)

  1. jsp3536 says:

    Where is the unit testing for app built with html and javascript?

  2. wpf_dev_no_more says:

    no thanks, looks really bad,

    exciting platform , no ugly kludge, you've ruined best dev platform. well done!

    upper case, wow, 5 years on WPF dev wasted, cya M$ , Scott Gu must be rolling over in his grave!

  3. Mike says:

    what about ARM?  only choice on the dropdown is remote.

  4. Anonymous says:

    Hi

    This is not relevant to building windows 8 apps, however with the surface rt and pro you are releasing will the stand on them be able to be layed down on a bench or desk so that the tablet is slightly raised to perform typing on the screen, or is the hinge on the upcoming devices not strong enough to support this position as well as its primary upright position.

    Thank you

  5. This still is quite of bit of information to digest if on is not familar with Visual Studio. I have been for quite some time, but I do ask is it better to test and design w/ 10 Ultimate or 12? (For reasons of better cooperation between the two)

  6. Anonymouse says:

    Now I understand better what wpf_dev_no_more commented: http://www.riagenic.com/…/960

  7. bye bye says:

    @wpf_dev_no_more

    I totally agree with you, they've killed Windows.

    and I know who is responsible, yes I'm looking at you Sinofsky.

    I hope you are happy, I'll have to learn Obj-C.

    Microsoft will collapse under the weight of its internal politics.

    I hope KBE William H. G. would come back as CEO.

  8. vs-dying-byebye says:

    forget about ms completely …

    ms great? no this is the worst plattform…

    wp7 is dying .. window8 will die …

    -> welcome mac/ios and android/ubuntu bb10

  9. Student says:

    JavaScript is bad and you should feel bad.

    If I wanted to use shitty web technologies I'd make a website, not a windows app.

    XAML > HTML && CSS

    C# > JavaScript

    I seriously hope Sinofsky gest fired over this.

  10. YOU MAKE ME SICK says:

    If you dislike windows 8 then don't come here and post your pathetic comments, you imbosiles, keep it to yourself.

  11. WINDOWS SURFACE PRO says:

    I hope the Surface PRO has excellent battery life like at the very least 8 hours. If not  i will sadly resort to other tablets from your oem partners like Samsung or lenevo that promise an excellent battery life of 10 hours. Please don't let me down let alone all the hundreds of people who I have read in blogs that are hoping for good battery life also. Please ensure the surface pro has an excellent battery life, please.

  12. WINDOWS SURFACE PRO says:

    WINDOWS SURFACE PRO Tuesday, August 28, 2012 5:18 PM #

    I hope the Surface PRO has excellent battery life like at the very least 8 hours. If not  i will sadly resort to other tablets from your oem partners like Samsung or lenevo that promise an excellent battery life of 10 hours. Please don't let me down let alone all the hundreds of people who I have read in blogs that are hoping for good battery life also. Please ensure the surface pro has an excellent battery life, please.

  13. WINDOWS 8 ROCKS says:

    You run the world MICROSOFT, you run the world MICROSOFT…..

  14. HASWELL - Surface Pro says:

    I hope the Surface Pro has the new Haswell chip from intel, as it would be the best tablet in the world. If it doesn't then myself and many others will wait for other tablets with haswell. Many people whose comments I have read are keen to wait for other tablets with Haswell, don't disappoint Microsoft or else you will lose these people.

  15. GT. says:

    Waiting for the next windows used to be so wonderful, reading all materials, convincing customers to upgrade and benefit from the innovations, …. And then came Windows 8 :(

    I am waiting for the day Chrome OS will install on normal PCs and able to run virtual machines, I need to use the old windows for a bit of time in VMs before I completely switch.

  16. LOL says:

    @GT

    You obviously have no clue about windows 8 and the vast majority of improvements this OS has. Gosh, i sure wouldn't want to purchase anything from your store as you have no clue about what you're talking about.

  17. JJ says:

    @ HASWELL – Surface Pro

    Same here man, gonna wait for tablets with haswell chips, so are my mates.

  18. Ravindra says:

    Is there a way to get the simulator as a standalone application i.e. out of Visual Studio, and launch apps there?

  19. Damien Cos says:

    @haswell – surface pro

    Yep I'm gonna wait for tablets with haswell too. Windows 8 pro tablets have poor battery life when they should be getting at least 10 hours, they are to thick, they are to heavy and consequently look and feel terrible in comparision to super thin tablets using atom chips.

  20. RIGHT ON says:

    I absolutely agree with your statements I am disappointed too!

  21. Damn says:

    WHY OH WHY MICROSOFT IS YOUR SURFACE PRO SO THICK. YOU CARRY ON IN YOUR KEYNOTE ABOUT HOW YOU CAN CRAFT METALS TO SUCH SMALL MEASUREMENTS SO WHY IS THE SURFACE PRO AT 13.5 MM  WHEN OTHER TABLETS SUCH AS SAMSUNGS ATIV SMART PC PRO ARE 11.89 MM AND HAS THE SAME SPECS AS YOURS.  WHY COULDLD YOU USE THE NEW ATOM CLOVER TRAIL CHIP OR THE HASWELL CHIP AND GET DOWN TO 9.9MM AS SHOWN BY SAMSUNGS ATIV SMART PC OR THE ASUS  ATIV PC. REALLY MICROSOFT REALLY? I REALLY WANTED TO PURCHASE A WINDOWS SURFACE PRO AND I WISHED IT DID WELL IN THE MARKET BUT NOW I THINK, WHAT A WASTE YOU HAVE MADE TO GET INTO THE TABLET MARKET. I AM REALLY REALLY DISSAPOINTED AND UPSET. YOU PRODUCT WAS GOING TO BE GOOD BECAUSE OF THE COOLING SYSTEM AND THE X2 ANNTENAS BUT NOW IM NOT GOING TO GET IT AS ITS TOO THICK, BATTERY LIFE WILL BE UNDOUBTEDLY POOR  AND EVEN THE TOUCH COVERS MOUSE PAD IS WAY WAY TO SMALL. I JUST DON'T GET IT. IF YOU ARE GOING TO DO SOMETHINGWHY NOT DO IT PROPERLY. TO ADD TO THAT YOUR OEM PARTNERS HAVE DONE A FAR BETTER JOB IN CREATING THEIR THIN AND LONG LASTING TABLETS. IM SO DISAPPOINTED, NO EXTREMLY  DISSAPOINTED. EVERYTHING I HAVE STATED HERE IS THE ABSOLUTE TRUTH AND I AM UPSET BECAUSE OF IT. BYE.

  22. DAMN says:

    WHY OH WHY MICROSOFT IS YOUR SURFACE PRO SO THICK. YOU CARRY ON IN YOUR KEYNOTE ABOUT HOW YOU CAN CRAFT METALS TO SUCH SMALL MEASUREMENTS SO WHY IS THE SURFACE PRO AT 13.5 MM  WHEN OTHER TABLETS SUCH AS SAMSUNGS ATIV SMART PC PRO ARE 11.89 MM AND HAS THE SAME SPECS AS YOURS.  WHY COULDLD YOU USE THE NEW ATOM CLOVER TRAIL CHIP OR THE HASWELL CHIP AND GET DOWN TO 9.9MM AS SHOWN BY SAMSUNGS ATIV SMART PC OR THE ASUS  ATIV PC. REALLY MICROSOFT REALLY? I REALLY WANTED TO PURCHASE A WINDOWS SURFACE PRO AND I WISHED IT DID WELL IN THE MARKET BUT NOW I THINK, WHAT A WASTE YOU HAVE MADE TO GET INTO THE TABLET MARKET. I AM REALLY REALLY DISSAPOINTED AND UPSET. YOU PRODUCT WAS GOING TO BE GOOD BECAUSE OF THE COOLING SYSTEM AND THE X2 ANNTENAS BUT NOW IM NOT GOING TO GET IT AS ITS TOO THICK, BATTERY LIFE WILL BE UNDOUBTEDLY POOR  AND EVEN THE TOUCH COVERS MOUSE PAD IS WAY WAY TO SMALL. I JUST DON'T GET IT. IF YOU ARE GOING TO DO SOMETHINGWHY NOT DO IT PROPERLY. TO ADD TO THAT YOUR OEM PARTNERS HAVE DONE A FAR BETTER JOB IN CREATING THEIR THIN AND LONG LASTING TABLETS. IM SO DISAPPOINTED, NO EXTREMLY  DISSAPOINTED. EVERYTHING I HAVE STATED HERE IS THE ABSOLUTE TRUTH AND I AM UPSET BECAUSE OF IT. BYE.

  23. Bob mars says:

    Waiting for tablets with  haswell too……waiting……

  24. ONLY LOGICAL says:

    People want a tablet that:

    – runs windows 8 pro

    – is powerful (fast and fluid)

    – has a long battery life (10+ hours of heavy use in real life)

    – is thin (9.9mm thin)

    – Is light (880 grams)

    If any company such as you MICROSOFT make a tablet with these features then everyone will buy your tablet, until then uh uh. That's why Im waiting for tablets with haswell as they will provide all these things, hopefully if the company does it right and doesn't cut corners.

  25. You sir are smart - no joke says:

    @logical

    You sure are correct in your statements. I am eagerly awaiting for tablets with haswell instead of the current ones on offer, as their to bulky and don't have a long battery life.