Anatomy of Test Automation “Tooling”

In many Microsoft product development teams, there are dedicated resources that work on test automation “tooling”.  For instance, I lead a team for Microsoft Dynamics AX that works on a portion of the test automation “tooling” for the team.  But what exactly is the “tooling” that is needed for successful test automation?  This post provides my perspective on the key components in the test automation “tooling” anatomy.

Here’s a simple picture of the anatomy that is typical of automation systems.  Some tooling, like the Visual Studio 10 Test and Lab Management toolset, has several or all of these components integrated.

Here’s a more detailed explanation of each box, with a references to Visual Studio 10 test tooling as examples in some cases.

· The System Under Test (SUT) in the green box isn’t part of the anatomy… it’s the thing you are testing!

· As you can see by the lines pointing to it, the Test Case Manager (TCM) contains core information that is used throughout the automation system.  Metadata about the test cases can come from attributes in the test cases, but also can get entered directly into the TCM.  The TCM has results connected to it so that a variety of reports can be performed.

· Automation developers most frequently interact with the frameworks used for developing test automation.  While the Test Source can be sliced in  number of ways, I like to think about the following three primary framework categories:

o The Test Harness provides the basic framework for writing the test code.  This includes a mechanism for specifying a test case, validation through assertion methods, and ways to aggregate and run tests.  In VS, the test harness is MSTest.

o Abstraction Frameworks provide a programming model for common operations in the test cases.  They are a key mechanism for eliminating code duplication, simplifying test code, and improving test code resiliency to product changes.  This is something that a “tools” team will frequently develop and is a core piece of successful automation.  Teams can get swallowed up by maintenance if investment is not made here.

o Interaction Frameworks enable test code to interact with the System Under Test for control and observation.  In VS10, I think of the “action recordings” and the tooling that is able to execute them using standard accessibility APIs as being an interaction framework. 

· The Test Execution Engine directs the test runs by allocating machines and distributing tests to them.  Checkout the Test Controller and the Lab Management Workflow in this VS10 Test Tools presentation. 

· Test execution produces Test Results & Logs.  The base test result has multiple states and the logs are essential for analyzing the test.

· Last but certainly not least, an Automation System needs to have effective Reporting & Analysis.  Doing all of the work to write test code and execute the tests goes to waste if we don’t act on the results.  See Tracking Test Results and Filing Actionable bugs in the VS10 Test Tools presentation.

Thinking about all of these pieces and how they work together gives some insight into why reliable test automation is such a big challenge to achieve! Investment in “tooling” for the components of the anatomy is the best way to tackle this challenge.