Code Coverage Collection

  • Article

By the end of my last post about code coverage instrumentation, we have instrumented our executable image and it's now ready for code coverage collection.  Code coverage collection is the process of collecting the runtime data of what functions were covered during test execution.   The end result of coverage collection will be a coverage data file which can be analyzed to show the coverage results to us in a meaningful way.

Visual Studio uses a separate tool to monitor, coordinate, and flush the coverage data to the coverage file.  This tool is VSPerfMon.exe - called the "Visual Studio Performance Monitor" (named so because it was first used by the profiler).  When the monitor is started, it listens for incoming connections from executables that have been instrumented for coverage.  One of the details I didn't mention about instrumentation is that when an executable image is instrumented, the instrumentation process creates a dependency on VSCover90.dll (9.0 for Orcas, 8.0 for 2005) for the image being instrumented.  This is accomplished either though the PE import table for native images or through an injected P/Invoke for managed images.  VSCover90.dll is essentially the code coverage runtime and it contains the logic for communicating with the monitor.  When VSCover90.dll is loaded and initialized, it will attempt a connection to the monitor.  If the monitor is not running, the connection will silently fail and coverage collection will be disabled for the lifetime of the process.  An instrumented image will execute normally without the monitor running.  If, however, the monitor is running and the connection attempt succeeds, VSCover90.dll will register the hosting process and will begin coverage collection.

VSPerfMon.exe can be found with the other profiler and code coverage tools in "%ProgramFiles%\Microsoft Visual Studio 9.0\Team Tools\Performance Tools".  To run the monitor as a foreground process, execute the following command from a command prompt:

 vsperfmon /coverage /output:<path_to_coverage_file>

This will run the monitor in "stand-alone mode".  In this mode you can see the processes that have connected to and disconnected from the monitor.  The monitor traps Control-C, so to shutdown the monitor you will need to use another tool called VSPerfCmd ("Visual Studio Performance Command").  This tool communicates with the running monitor process to perform a variety of tasks.  You would execute the following command line to shutdown the monitor:

 vsperfcmd /shutdown

The command tool can also be used to start the monitor in the background.  To do so, execute the following command line:

 vsperfcmd /start:coverage /output:<path_to_coverage_file>

This command will cause the monitor to be spawned as a background process.  You will use the same command as the one given above to shutdown the monitor.

There are a few important things to note about the monitor that can prevent successful coverage collection:

  • By default, the monitor only accepts connections from processes running under an identity that is a local administrator.  The most common problem experienced with this is that no data is collected for ASP.NET web sites.  This is due to ASP.NET web sites running under the Network Service identity by default, which is a restrictive identity meant for services that accept network connections.  To solve this, there is a /USER switch on VSPerfMon/VSPerfCmd to allow the specified user access to the monitor.  When you perform code coverage on a web site inside Visual Studio, a request is made by Visual Studio to discover what the identity of the worker process is.  This identity is then passed to the monitor to allow the connection.  This happens automatically before test execution when there is a web site checked in the code coverage section of the test run configuration.
  • By default, the monitor creates the objects and communication channels needed by the coverage runtime in a local session scope.  Therefore, if you are running the monitor in a different session from the process that will connect to the monitor (e.g. if you are not logged on from the console session and you want a service to connect to the monitor), then you need to specify the /CrossSession switch to VsPerfMon/VsPerfCmd.  This will instruct the monitor to create those objects in a global scope so that they are accessible from other sessions.

Let's recap what we have so far:

  • Instrumentation is performed using vsinstr.exe for native, mixed mode, and managed images.  To instrument an image, execute the following: "vsinstr.exe /coverage <path_to_image>".
  • To start the monitor process that will collect the coverage data, execute the following: "vsperfcmd /start:coverage <path_to_coverage_file>".
  • Run the test(s) you want to collect coverage data for using the instrumented images.
  • To stop collection, execute the following: "vsperfcmd /shutdown".

We now have a coverage file which we can use to view coverage results.  The coverage file can be imported into the "Code Coverage Results" tool window inside of Visual Studio.  This will show you per-method statistics and double clicking on a method will result in opening the source file in the editor with coverage coloring turned on.  I will present a walk-through of how to accomplish everything I've been talking about so far inside of Visual Studio in a future blog entry.

I will discuss how to use the coverage analysis API to read the coverage file programmatically in my next blog entry.