Visual Studio Test Platform – upcoming changes to data collectors

Pratap Lakshman

Having executed a test – at a minimum – we want to know the outcome. But beyond that, we may want to know more. Data collectors and loggers are the key extension mechanisms intended to provide this and other such details for rich reporting.

Data collectors and loggers generate information based on listening to events raised during test execution. A logger might emit simple outcome information (that is the console logger used by vstest.console.exe), while a “trx” logger would emit additional information as a .trx file (/logger:trx). Data collectors are a little more advanced. A “Code Coverage” data collector would collect and emit code coverage information as a .coverage file. A “video” data collector on the other hand would emit a screenshot capturing the screen contents at the moment a test completed execution. Why, even Test Impact Analysis uses a TIA data collector!

There are several aspects to consider when working with data collectors (1) Authoring: the involves authoring a data collector based on the relevant vstest extensiblity points. (2) Packaging: how should the data collector be packaged? As a vsix? As a NuGet package? As a zip file? (3) Identifying: how do you identify a data collector with vstest? Is it by the name of its binary? Or, by the full path to the binary? Is it with a Uri? (4) Enabling: how do you indicate to vstest to hook up the data collector during execution so that it may listen to events? Is it via a command line switch? Is it via a .runsettings file? (5) Configuring: there are some data collectors that may support additional configuration. For instance, a code coverage data collector might support configuring to collect data for only some named set of binaries. (6) Execution: does the data collector run in-proc with vstest? Does it run in a separate process? What are the implications? (7) Discovery: where does vstest probe when looking for a data collector? Where does it probe in the context of running from Visual Studio? Where does it probe when running from the command line? Or from the VSTest task in CI? This is closely related to (2) and (3).

Migrating data collectors If you are writing data collectors these are important considerations. Especially so now. Here is why – in an earlier post you would have noticed vstest.console.exe being used from the following 2 locations:

C:\Program Files (x86)\Microsoft Visual Studio\...\IDE\CommonExtensions\Microsoft\TestWindow
C:\Program Files (x86)\Microsoft Visual Studio\...\IDE\Extensions\TestPlatform

vstest.console.exe in the TestPlatform folder is the version that is being actively developed in the repo https://github.com/Microsoft/vstest. It supports .NET Core, is cross-platform (Windows, Linux, Mac) and OSS, as explained in this series of blog posts. It will eventually wholly replace the version found in the TestWindow folder. It introduces differences and enhancements to the data collection infrastructure. You can read more about writing a data collector here: DataCollector. If you already have written data collectors, then you will need to migrate them. The migration steps are straight forward and called out in the migration guide posted here: data collector migration guide.

Go ahead, read the document, and migrate your data collectors. Looking forward to hearing your feedback!

Pratap Lakshman Senior Program Manager

Follow