Testing with Visual Studio Team System
I recently attended and presented at both TestExpo and the Software & Systems Quality Conference and at both events people seemed surprised to see a Microsoft stand. In my experience testers don’t seem to be as aware of our tools as developers so hopefully I introduced a few more people to them at these events. I promised to put together a blog posting laying out what Visual Studio Team System is all about and what should testers really should know about it?
Visual Studio Team System is really broken into two parts: Team Foundation Server and Visual Studio Role Editions.
Team Foundation Server
Team Foundation Server (TFS) is an integrated collaboration server to help you drive your development process from requirements through to deployment. It allows you to store all project artefacts in one location and access that information and data from a tool of your choice (eg. Visual Studio, Outlook, Microsoft Project, Excel, Eclipse, Web Access, SharePoint etc)
- Source Code Repository
- TFS is an enterprise quality Version Control for all source code (not just .Net). TFS is NOT based on Visual SourceSafe
- Project Portal
- TFS provides a Windows SharePoint Services Portal for every project to give one central location for project documentation
- Work Item Tracking
- Work Items give visibility of all work outstanding in your project (eg. use cases, development tasks, test cases and bugs). Work Items are fully customisable
- Project Reporting
- SQL Server Reporting Services make it simple to produce high quality reports to provide better project transparency.
- Build Server
- TFS offers a tightly integrated and highly configurable build component based on MSBuild
- Supports your Development Process
- TFS is process agnostic, and can be configured to support any process with a Process Template .
Visual Studio Team System Role Editions
The 4 (or really 3 now) Visual Studio Team System Role Editions are:
- Development
- This Edition is focused on code quality.
- Database
- This edition has now been rolled into the Development Edition so will no longer be a stand alone product. It provides a way to improve quality and change control at the data layer.
- Test
- Providing web testing, load testing and manual testing tools.
- Architecture
- Improves the design and validation of distributed systems giving architects and developers the ability to visually construct service-oriented solutions and validate them against their operational environments prior to deployment.
- Suite
- Includes all functionality provided by the role editions
All VSTS Role Editions are built on top of Visual Studio Professional Edition, so all features that are part of VSPro are included in the VSTS roles alongside their extra functionality
So what do Testers really need to know?
So, as this post is aimed at testers, what should testers be aware of in the current version of VSTS and also what is coming down the line in VSTS 2010 (codename Rosario)?
Visual Studio Team System 2008
Quality Assurance
Testing and maintenance are expensive so teams need to ensure that the builds that are actually being tested are of good quality. The only way to do this is to encourage your developers to think about quality from day one!
Unit Testing
The VSTS Unit Testing Framework has been pushed down to VSPro so the tools are there for all developers to write unit tests. Code coverage of these tests can also be measured to give more confidence on code quality
Database Unit Testing
Unit Test your stored procedure to ensure quality at the data layer as well as the code layer
Static Code Analysis
You can think of Static Code Analysis as a first pass automated code review. Code Analysis will flag up many common issues which can be difficult to spot or tedious to find.
Guidelines for Using Code Analysis Tools
Code Metrics
How Complex is your code? How difficult will it be to test properly? How maintainable is it? These are the kind of questions that can be easily answered through the code metrics. They will give the developers a heads up on where they should be concentrating their code reviews to ensure quality, maintainable applications.
Profiling
How does the application actually perform? VSTS Profiling tools will enable developers to locate and eliminate bottlenecks in their code to ensure that the applications perform as expected
Analyzing Application Performance using Profiling Tools
Check In Policies
Essentially Check-In Policies are a quality gate on version control, tasks that must be completed before the code can be checked in and built. Examples are the testing policy to ensure regression bugs are not introduced by a code change, the code analysis policy to ensure checked in code meets certain agreed standards and the Work Item policy to reduce scope creep and help testers to pinpoint what has been included in any given build of the application.
Continuous Integration & Automated Builds with Build Verification
TFS makes it very simple to set up automated builds. It is easy to include build verification tests and code analysis as part of these automated builds but they can be further extended to create installation packages, deploy test data etc.
Development Process & Defect Tracking
TFS can assist teams in adopting a development process or simply encourage them to take a step back and understand how they can improve their in-house (possibly undocumented) process. By storing requirements, development tasks, test cases, builds and test results in one central location and allowing links between them, teams can gain much better visibility of project health and improve communication between all members of the team.
Team Foundation Server Reporting
Take advantage of the power of TFS reporting to ensure your project is on track! As everything related to your project is stored in TFS, all of that data is collected into the data warehouse. Use SQL Reporting Services or Microsoft Excel to access the data you are interested in.
Testing
As I mentioned earlier in the post, many people are simply unaware of the testing tools included in VSTS. These tests can be authored by non-technical testers but can be enhanced and extended through code if required.
Web Testing
Web tests are used to automate testing the functionality of Web applications and to see how they perform under load. Record your interaction with a website and play it back.
Load Testing
Bringing load testing to the masses! The primary goal of a load test is to simulate many users accessing a server at the same time.
Manual Testing
Manual Tests can be executed within Visual Studio like any other test. Once the tester has followed the instructions in the manual test script they can add their feedback and apply a test pass or fail. This allows you to report against your manual test results and keep track of build quality.
Data Generation
Testing with live data can be dangerous so if you are using SQL Server as your data store then Data Generation plans allow you to generate meaningful data the same size and shape as your live data.
Test Case Management
Test cases can be managed through the use of TFS Work Items and Test Lists, test results can be published so they can be reported against and test failures can be attached to bugs which are then assigned to the dev team
Work Item Only View
Team System Web Access provides testers or users a “Work Item Only” view that restricts functionality so that you can create and view only your own work items. This means that you can submit defects and change requests directly to the dev team without the need for a Team Foundation Server Client Access License. For more information, see Visual Studio Team System 2008 Licensing White Paper.
Visual Studio Team System 2010
Some of the manual testers I spoke to in the last few weeks were a little daunted by Visual Studio. They don’t write any code and Visual Studio Test Edition contains a lot of functionality that they don’t need. Well Visual Studio Team System 2010 is going to address these concerns and add a bucket load of cool features around testing
More details to follow when I get a chance to digest all the new stuff myself but keep an eye out for:
- Test planning tools to breakdown requirements, plan tests and test configurations and assign testing responsibilities.
- A TFS based test case management system that enables you to track all of your test cases, relate them to requirements and plans, track progress and perform impact analysis.
- Test prioritization that highlights the tests that need to be run based on the changes that have actually been made to an application.
- A tool for manual testers to help them manage their test runs, automate them to reduce manual labour and report actionable bugs that the developer is sure to be able to reproduce and fix.
- Tools for developers to see exactly what testers have done when finding a bug, including the ability to "debug" the code as it was running at the time they hit the bug.
- A new "gated checkin" feature which provides a natural evolution of continuous integration to catch build breaks before they happen rather than after.
- And much more...
(Thanks to Brian for the list)
If you need anymore information on testing with VSTS then check out the links below or Contact me
Resources
Videos: Getting Started with Team System
Getting Started with Team System Testing Tools
"How Do I?" Videos for Team Foundation Server
Visual Studio 2005 Team Test Edition: Testing Demos
These demos show the unit and load testing features in Visual Studio 2005 Team Test Edition which are similar to the 2008 features
Brian Harry (Probably the best Team System blog around)
Changing a Web Vanguard – Windows Live Hotmail
This case study documents how the Windows Live Hotmail development team successfully utilized Visual Studio Team Edition for Software testers to build one of the most ambitious new releases of their iconic Web-based email service.
March Madness and Visual Studio Team Edition for Software Testers
This case study illustrates how the Windows Live Development team was able to design, build and deliver a high profile consumer facing Web application under extremely tight deadlines by making use of Visual Studio Team Edition for Software Testers.
Happy Testing!
Richard Erwin