This is the first of a series of posts on PowerShell Core and the tools we use to test it. If you’ve looked at the main project for PowerShell (https://github.com/PowerShell/PowerShell, you may have noticed a new badge down in the Build status of nightly builds:
We are supplying code coverage numbers for our test pass via the OpenCover project (https://github.com/OpenCover/opencover) and we visualize our code coverage percentage via coveralls.io (https://coveralls.io/github/PowerShell/PowerShell?branch=master). This means you can see some details about our testing and how much of PowerShell is covered by our test code.
You can get your own coverage numbers easily via our OpenCover module which may be found in the <RepoRoot>test/tools/OpenCover directory. To generate a code coverage report, you need to create a build which supports code coverage. Currently, that’s only available on Windows, but we do have an easy way to get it:
(All of these commands assume that you are at the root of the PowerShell repo)
# create a code coverage build. PS> Start-PSBuild -Configuration CodeCoverage -Publish # Now that you have a build, save away the build location PS> $psdir = split-path -parent (get-psoutput) # Import the OpenCover Module PS> Import-module $pwd/test/tools/OpenCover # install the opencover package PS> Install-OpenCover $env:TEMP # now invoke a coverage test run PS> Invoke-OpenCover -OutputLog Coverage.xml -test $PWD/test/powershell -OpenCoverPath $env:Temp/OpenCover -PowerShellExeDirectory $psdir
If you want to get code coverage for only the tests that we run in our Continuous Integration (CI) environment, add the parameter -CIOnly. Then you’ll need to wait for a bit (on my system and using -CIOnly, it takes about 2.5 hours to run).
Looking at the Data
The OpenCover module can also help you visualize the results from a very high level.
# first collect the coverage data with the Get-CodeCoverage cmdlet PS> $coverData = Get-CodeCoverage .\Coverage.xml # here’s the coverage summary PS> $coverData.CoverageSummary NumSequencePoints : 309755 VisitedSequencePoints : 123779 NumBranchPoints : 105816 VisitedBranchPoints : 39842 SequenceCoverage : 39.96 BranchCoverage : 37.65 MaxCyclomaticComplexity : 398 MinCyclomaticComplexity : 1 VisitedClasses : 2005 NumClasses : 3309 VisitedMethods : 14912 NumMethods : 33910 # you can look at coverage data based on the assembly PS> $coverData.Assembly | ft AssemblyName, Branch, Sequence AssemblyName Branch Sequence ------------ ------ -------- powershell 100 100 Microsoft.PowerShell.CoreCLR.AssemblyLoadContext 45.12 94.75 Microsoft.PowerShell.ConsoleHost 22.78 23.21 System.Management.Automation 41.18 42.96 Microsoft.PowerShell.CoreCLR.Eventing 23.33 28.57 Microsoft.PowerShell.Security 12.62 14.43 Microsoft.PowerShell.Commands.Management 14.69 16.76 Microsoft.PowerShell.Commands.Utility 52.72 54.40 Microsoft.WSMan.Management 0.36 0.65 Microsoft.WSMan.Runtime 100 100 Microsoft.PowerShell.Commands.Diagnostics 42.99 46.62 Microsoft.PowerShell.LocalAccounts 0 0 Microsoft.PowerShell.PSReadLine 6.98 9.86
I’m not going to go through all the different properties that are reported, we’ll take a closer look at those in future posts. The Get-CoverageData cmdlet is still fairly rudimentary, but it will provide some details. This is part of our public repo, so I encourage you to enhance it and log issues if you find them!
Better Coverage Visualization
Another way to view coverage data is via the ReportGenerator package, which creates HTML reports and provides much more details about the coverage. The ReportGenerator package is available via the find-package cmdlet in the PackageManagement module. The following will install the package, and show how to run it:
# find and install the report generator package PS> find-package ReportGenerator -ProviderName nuget -Source https://nuget.org/api/v2 | install-package -Scope CurrentUser PS> $ReportGenExe = “$HOME\AppData\Local\PackageManagement\NuGet\Packages\ReportGenerator.2.5.2\tools\ReportGenerator.exe” # invoke the report generator and create the report in c:\temp\Coverage PS> & $ReportGenExe -reports:Coverage.xml -targetdir:c:\temp\Coverage
Now that you’ve created the reports, you can visualize them with your browser.
PS> invoke-item C:\temp\Coverage\index.htm
Click on the “Enable filtering button”, and then “Collapse all” and you should see something similar to:
You can then drill in on what interests you (Microsoft.PowerShell.Commands.Utility, for example)
Of course, there’s a lot more detail to discover, and I encourage you to poke around. In my next post, I’ll go through an entire workflow:
- Select an area for improvement
- Create new tests
- Gather new coverage data
- Compare results from previous runs
I’ll target something specific (a cmdlet) and show how to determine the gaps and how to fill them.
Call To Action!
Now that you see how easily you can generate code coverage data, this is a great opportunity to provide some additional coverage and increase the quality of our releases. If you see some area which you’re passionate about or notice an area which you would like to measure better, it’s a great way to provide improved coverage. As you create new PRs, you can aim for high coverage in your new functionality (85-90%), and now you can measure it!