Continuous Testing in DevOps
In older testing strategies, large software changes were tested as a complete product after a so called “release to QA”, running almost all tests just before release. We know the downsides to that. On the other hand, DevOps is all about a fast development to delivery pipeline and continuous delivery of value. Releases are happening in days and weeks – not in years as they used to. In such a DevOps world, there is not going to be any continuous delivery if you do not have your testing right. Just as we have CI Continuous Integration (CI) and Continuous Delivery (CD), DevOps calls for Continuous Testing (CT).
Test Impact Analysis (TIA)
Can such continuous testing also be fast? Can it still be comprehensive? Should you just run all your tests all the time? At smaller scales running through all your test suites and emitting out copious test results may be tolerable, but this not scale. At a larger scale, for testing cycles to be fast, a more sophisticated view of what is comprehensive testing becomes essential. “Relevance” becomes critical – only run the most relevant tests and only report the relevant results.
The Test Impact Analysis (TIA) feature specifically enables this – TIA is all about incremental validation by automatic test selection. For a given code commit entering the pipeline TIA will select and run only the relevant tests required to validate that commit. Thus, that test run is going to complete faster, if there is a failure you will get to know about it faster, and because it is all scoped by relevance, analysis will be faster as well.
TIA does come with a little overhead of its own (in order to build and maintain the mapping), and so is best applied only in cases where a test run itself takes a little long to complete (say, > 10 mins).
TIA is supported via the Version 2.* (preview) of the Visual Studio Test task. If your application is a single tier application, all you need to do is to check “Run only impacted tests” in the task UI. The Test Impact data collector is automatically configured. No additional steps required.
If your application interacts with a service in the context of IIS, you need to additionally configure the HTTP Proxy data collector and the Test Impact data collector to run in the context of IIS (both, via the .runsettings file). Here is a sample .runsettings file that does this configuration: samplerunsettings.
TIA is in integrated into existing test reporting at both the summary and details levels, including notification emails.
Robust Test Selection
But why should you trust TIA? How do you know that quality has not been compromised in all of this? To answer that we need to look at 2 aspects:
(1) what are the tests that get automatically selected
(2) policies that can condition test selection.
TIA will look at an incoming commit, and select the set of relevant tests – these will have 3 components
(1) The existing tests impacted by the incoming commit.
(2) Additionally, it will also select previously failing tests – if not, then over the course of several commits some earlier failing test case might just get lost … Therefore, TIA will keep track of tests that failed in the previous build and include them in the selection.
(3) It will also include newly added tests – what if your commit contains new tests? Such tests could uncover product bugs, right? So, TIA will select newly added tests as well.
Taken together, this complement makes for a robust selection of tests. Additionally, we allow (and recommend) running all tests at a configured periodicity.
TIA is presently scoped to the following:
- Requires use of the Version 2.* (preview) of the Visual Studio Test task in the build definition.
- Requires either VS2015 Update 3 onwards OR VS 2017 RC onwards on the build agent
- Managed code
- CI and in PR workflows
- IIS interactions
- Automated Tests (tests and application must be running on the same machine).
- Build vNext workflow, with multiple VSTest Tasks
- Local and hosted build agents
- Git, GitHub, External Git, TFVC repos
- Not yet supported
- Remote testing (where the test is exercising an app deployed to a different machine)
- xplat (Windows only).
- UWP support.
We will continue this series in the next post, looking at more details, specifically: manual overrides, repo considerations, using TIA in PR builds, and interaction with Code Coverage. If you would like any other aspect covered in more detail, let us know.
In the meantime, start using TIA! Looking forward to your feedback.