Testing for Continuous Delivery with Visual Studio 2012 RC

  • Article

clip_image001

Customer Value

Testing has changed dramatically over the last several years, and what we've learned has changed our tools and our workflows. While the goal of a testing organization is to continuously test a system to ensure the best experience for its customers, the reality is bit different.  Because of the cost and pain involved in building a test infrastructure, continuous delivery has not traditionally been achieved.

Testing has always been the less glamorous sister to software development, scarcely noticed outside the industry, and the object of complaints inside. If some small error is missed, testing is to blame; if time is running short or costs are getting too high, testing is likely to be starved of resources. When combined with these attitudes, the dependency on the existing test infrastructure means that the idea of changing any part of the testing process makes engineers and managers drag their feet unless they can clearly see the value.

This guide addresses the costs and pain points traditionally involved in testing by contrasting the more conventional test approach employed by Contoso with the new approach Fabrikam takes using the Visual Studio 2012 RC testing infrastructure.

In interviews with a number of customers, we learned that many weren't taking advantage of all the features of Lab Manger and of the other critical features of the test infrastructure, and many lacked an understanding of the value proposition of such an approach and how it can benefit their overall projects.

The Visual Studio 2012 RC application lifecycle management testing infrastructure can help customers create a continuous testing solution for any software system they are building. Team Foundation Server serves not only as a source code repository, but when combined with the Build Controller, Test Controller, SQL Server, Virtual Memory Manager, and Visual Studio Test Manager, it represents a powerful set of tools for achieving continuous testing. Together these tools help teams reduce the pain and expense of aligning test cases with code coverage and automating regression testing.

This guide walks through the various scenarios and shows the reader how to achieve their most ambitious goals using Visual Studio 2012 RC tools.

Testing what would previously have taken weeks can be accomplished in hours thanks to the use of virtual environments for testing. Aligning test cases to code coverage becomes very straightforward with the test plans implemented in Test Manager. In addition, the process outlined here provides guidance on other types of tests, such as performance tests and manual regression testing to eliminate the issue of the non-reproducible test.

Finally, the guide takes the reader through various scenarios and helps provide insight into how to add value to their engineering workflow by using the various reporting tools.

What’s in Testing for Continuous Delivery with Visual Studio 2012 RC?

Component

Description

The Guide

Chapter 1: The Old Way and The New Way

This first chapter defines the context of the story, which centers around the merger between Contoso and Fabrikam, representing the tension between the more classical approach to software testing and the modern, agile approach.  This chapter makes the argument for why the approach to testing must change, and enumerates all the benefits to be gained.

Developers create and run unit tests using Visual Studio. These tests typically validate an individual method or class. Their primary purpose is to make sure changes don't introduce bugs. An agile process involves the reworking of existing software, so you need unit tests to keep things stable.

The next sort of test you need is the system test. To test a system, you must first install it on a suitable machine or set of machines. In Visual Studio, lab environments take a lot of the tedium out of setting up fresh computers and configuring them for testing. Typically this is done via virtual machines, which provides many benefits including that setting up a new environment can be done very quickly by replicating a template.

System tests make sure that the software you are developing meets the needs of the stakeholders. System tests look at what you can do and see from outside the system—that is, from the point of view of users and other systems that are external to yours.

System testing is performed with Microsoft Test Manager. As well as planning tests and linking them to requirements, Microsoft Test Manager lets you set up lab environments on which to run the tests.

System testing starts with exploration—just probing the system to see what it does and looking for vulnerabilities ad hoc. But gradually you progress to scripted manual testing, in which each test case is described as a specific series of steps that verifies a particular requirement. This makes the tests repeatable; different people can work through the same test without a deep understanding of the requirement and reliably obtain the same result.

In addition, manual tests can be made faster and easier by recording the actions of the first tester, and then replaying them for subsequent tests.

Functional tests are just the beginning. You'll want to do load tests to see if the system can handle high volumes of work fast enough; perform stress tests to see if it fails when it falls short of memory or other resources; and do security, robustness, and a variety of other kinds of tests.

Visual Studio has specialized tools for some of these test types, and in others there are testing patterns we can recommend.

Whether you're aiming for deployment ten times a day, or you just want to reduce the cost of running tests, it isn't just a matter of running the tools. You have to have the right process in place.  Different processes are appropriate for different products and different teams. Continuous delivery might be appropriate for a social networking website, but not less so for medical support systems.

Whether your process is rapid-cycle or very formal, you can lower the risks and costs of software development by adopting some of the principles of agile development, including rigorous testing and incremental development.

If you're administering your test framework, the appendix is for you. We walk through the complete setup and discuss your options. If you follow it through, you'll be ready to hire a team and start work. (Alternatively, a team will be ready to hire you.)

We put this material at the end because it's quite likely that someone else has already done the setting up, so you can dig right into testing. But you'll still find it useful to understand how the bits fit together.

The bits we install include: Visual Studio Team Foundation Server and its source and build services; Microsoft SharePoint Team Services, which provides the project website on which reports and dashboards appear; Microsoft Hyper-V technology and Microsoft System Center Virtual Machine Manager (SCVMM).

The companion samples

The samples illustrate all scenarios covered in the book. They provide a “single box” experience with minimal infrastructure requirements.

Click here to download this release.  Click here to see this guidance on MSDN.

This documentation is based on prerelease software. It is for preview only, and is subject to change.

Goals of This Release

This book provides the information needed to set up a complete Visual Studio 2012 RC test infrastructure and provides working scenarios to illustrate the value that can be gained through the use of these tools as compared to taking the traditional approach to testing. It is intended for any tester, developer, information technology (IT) professional, or manager who must deal with any type of test infrastructure.

In addition, the book focuses on the various pains experienced when dealing in traditional testing approaches.

Acknowledgements

We want to thank the customers, partners, and community members who have patiently reviewed our early content and drafts. Among those, we want to highlight the exceptional contributions of Tiago Pascoal, Paulo Morgado, Debra Forsyth, Carlos dos Santos, Richard Hundhausen, Paul Glavich, Mike Douglas, Jacob Barna, and Hideaki Azuma. Very special thanks to Katrina Lyon-Smith - MSDN, Willy-Peter Schaub from the Rangers, Anutthara Bharadwaj and Muthukumaran Kasiviswanathan from the Dev. Tools Team for their continued support. We’d also want to thank Sam Guckenheimer & Björn Rettig, executive sponsors of this project.

About patterns & practices

The Microsoft patterns & practices (p&p) team is responsible for delivering applied engineering guidance that helps software architects, developers, and their teams take full advantage of Microsoft platform technologies in their custom application development efforts.

Our goal is to help software development teams be more successful with the Microsoft application platform. We do this by delivering guidance that:

· Helps to simplify the Microsoft application platform.

· Provides solution guidance to common problems.

· Helps development teams grow their skills and learn.

For more information: https://msdn.microsoft.com/practices or https://practices