Accelerated Continuous Testing with Test Impact Analysis – Part 4

Pratap Lakshman

August 4th, 20170 0

Essential to TIA’s test selection is the map of dynamic dependencies between test methods and source files of code exercised during their execution. TIA needs dependencies mapped in this form:

    TestMethod1
        dependency1
        dependency2
    TestMethod2
        dependency1
        dependency3

TIA can generate such a dependencies-map for managed code execution. Where such dependencies reside in .cs and .vb files, TIA can automatically watch for commits into such files and then run tests that had these source files in their list of dependencies. Out-of-the-box, that, in a nutshell, is TIA’s scope. But by explicitly providing the dependencies-map, TIA can be extended even beyond!

Extending TIA to new scenarios You can provide TIA the dependencies-map explicitly in an xml file. The mapping can even be approximate, and the desired tests-to-run can be specified in terms of a test case filter you would typical provide in the VSTest task.

TIA can thus be extended to scenarios not supported out-of-the-box. For e.g.:

To support code in other languages – JavaScript, C++, etc.
To support the scenario where tests and product code are running on different machines.

Here is an example – consider code organized in the repo: https://github.com/Microsoft/testfx. This is the repo for MSTest V2. The folder test/UnitTests/MSTest.Core.Unit.Tests/Attributes contains tests for various attributes, and these tests are enclosed in the namespace Microsoft.VisualStudio.TestPlatform.TestFramework.UnitTests.Attributes The corresponding product code is in the folder src/TestFramework/MSTest.Core/Attributes/

Based on this, you might want to run the attribute tests whenever there was a commit into any of the files in above folder. Remember the dependencies mapping that you specify can be approximate – therefore it can be specified as follows:

<TestImpactMap>
 <Tests>
  <Test Filter="FullyQualifiedName~AttributeTests">
   <Dependencies>
    <Dependency Name="/src/TestFramework/MSTest.Core/Attributes/**/*" />
   </Dependencies>
  </Test>
 </Tests>
</TestImpactMap>

Create an XML file with the above information, check it in. Assuming the file is checked in at the root of the repo, and is named TIAmap.xml, TIA can be pointed to this file by setting the build variable tia.usermapfile to $(System.DefaultWorkingDirectory)/TIAmap.xml

Now, when a commit comes in to any file in the dependent folder, TIA consults the mapping and runs only the test methods matched by the filter criteria specified!

In this case the dependency was mentioned at the level of a folder. It could as well have been at the level of a file, and that file could as well have had any extension, or could have been just an extension.

In effect, TIA can thus be extended to any file (.cpp, .js, or .xml files, etc.), and such files need not even contain executable code (as in the case of .xml files) and other languages, and to the cases where the tests and product code are running on different machines!

Go ahead, try it out on your repo, and let us know. We look forward to your feedback.