Accelerated Continuous Testing with Test Impact Analysis – Part 2
The previous post introduced how – for a given code commit – TIA will select and run only the relevant tests required to validate that commit. Thus, without sacrificing quality, both the testrun and its enclosing CI definition will complete faster. Here is how that translated to reality for one of our teams:
The top graph plots the time-to-completion for the CI definition
The blue curve is the observation when TIA is OFF. No test selection in effect, and hence “all” tests are run after each build – thus, the almost constant time taken for each run of the CI definition.
The orange curve is the observation when TIA is ON. Automatic test selection is in effect. Since only impacted tests are selected to run, the overall CI definition completes faster.
The bottom graph plots the testrun time.
The blue curve is the observation when TIA is OFF. No test selection in effect, and hence “all” tests are run – thus, the almost constant time taken for each testrun as well.
The orange curve is the observation when TIA is ON. Automatic test selection is in effect. Since only impacted tests are selected to run, the overall testrun time is predominantly lower.
Of course, YMMV.
Enable TIA in CI/PR workflows
Continuous Integration (CI) is a key practice in the industry. Integrations are frequent, and verified with an automated build that runs regression tests to detect integration errors as soon as possible. As a code base grows and matures its regression test suite tends to grows as well – to the extent that running a full regression test might require hours – slowing down the frequency of integrations, and ultimately defeating continuous integration. In order to have a CI definition that completes fast, some teams defer the execution of their longer running tests to a separate stage in the pipeline. This only serves to further defeat continuous integration.
DevOps is all about a fast development to delivery pipeline and continuous delivery of value. Releases are happening in days and weeks. In such a DevOps world, there is not going to be any continuous delivery if you do not have your testing right. There is no way that we are going to have development completed and integrated unless we also have a way to ensure that the development is ready to be entered into an integration pipeline.
TIA can rescue this situation. TIA is supported in the CI and PR workflows. TIA will automatically select only the subset required to validate the code being committed.
(1) a robust test selection – this includes existing impacted tests, previously failing tests and newly added tests.
(2) safe fall back – for commits and scenarios that TIA cannot reason about, it will safely fall back to running all tests (you can see that mentioned in the logs). TIA is currently scoped to only managed code, and single box topology. So for e.g. if the code commit contains changes to HTML/CSS files, it cannot reason about them and will fall back to running all tests.
(3) configurable overrides – you can run “all” tests at a configured periodicity.
Now you can, and ought to, make more of your automated tests available to run early in the integration pipeline.
Enable TIA. You have everything to gain.
A couple of caveats when using TIA with Visual Studio 2015 to be aware of:
(1) Running tests in parallel – in this case, tests will run serially.
(2) Running tests with code coverage enabled – in this case, code coverage data will not get collected.
We will continue this series in the next post, looking at more details, specifically: manual overrides, and repo considerations. If you would like any other aspect covered in more detail, let us know.
Looking forward to your feedback.