Testing a Team Services extension and tracking code coverage

Jasmine is a framework that allows you to write tests in JavaScript for AngularJs or NodeJs applications. In our case, we will use Jasmine to write tests for VSTS extension.

The Jasmine Framework is a Jasmine.js library. To retrieve it, we have two possibilities, either with a package Nuget or with an npm package.
In case the extension is developed with Visual Studio, we retrieve the Chutzpah package with Nuget that contains jasmine and its test adapter.

With Visual Studio Code (or with another IDE), we can retrieve this framework with npm with the command.

npm install --save-dev jasmine

If you are using Josh Garverick’s Visual Studio Project Templates, all components required to run Jasmine tests are already included in the npm packages configured in the "packages.json" file.

Writing a Jasmine unit test is fairly simple. The Project Template has an example in the test folder in the "testSpec.js" file.

Here is another example:

There are several ways to run Jasmine unit tests. With Visual Studio, you can use a test adapter like Chutzpah, which is one of the best known.  It is installed with Nuget by retrieving the latest version of the Chutzpah package.

Next, to be able to visualize and execute the tests in the Visual Studio Test Explorer, install an extension to Visual Studio called Chutzpah Test Adapter for the Test Explorer. Additionally, install the Chutzpah Test Runner Context Menu Extension to be able to execute tests from the contextual menu.

Once installed, the tests appear in the explorer test, and they can be run as unit tests.

You can use the grunt-contrib-jasmine grunt already existing in the template, and in the "gruntfile.js" file have the jasmine task.

Run grunt jasmine

To be able to set Chutzpah parameters, we add a configuration file json (at the root of the project) named "Chutzpah.json". Inside, we specify the file of specs to be executed, the mode of compilation of the typescript files, and the parameters of execution of the code coverage.

For all details on this file, see https://github.com/mmanela/chutzpah/wiki/Chutzpah.json-Settings-File

IMPORTANT - The following section contains paths as part of the configuration. These are solution-specific and likely to differ in your environment. Use the screenshots and examples as reference only! 

Set parameters

1. Execution Options - The tests path and pattern.
2. Advanced Execution Options - The root of test adapter : the packages folder where the Nuggets packages, including the Chutzpah package, are downloaded.

After the build, the results of the tests are obtained in the Summary tab.

And the details on the tests tab.

The code coverage indicators provide an indication (in percentage) of the number of lines covered by the tests.

One of the advantages of Chutzpah is that it allows this analysis and generates a report during the tests execution.
To activate it, add the parameters in the configuration file of chutzpah.

1. Enable the code coverage analysis and the files to includes and excludes.
2. Specify the file type for the analysis result; in our case, we will choose the lcov format.

With this configuration, the code coverage analysis will be done automatically at the same time that the tests will be executed, so there is nothing more to configure in the build.

After the build runs, the result of the analysis is visible in the summary tab.

Having this percentage is a good indication, but it's better to have a detailed report that indicates on each file the lines that are not covered by the tests in order to improve them.

In the result of the VSTS builds, the code coverage tab allows you to display this report, which is in the Jacoco or Cobertura format.

The problem we have is that the report generated by Chutzpah is in lcov format, so in the build execution convert this report into a report summary of type Cobertura, then, in html format, and, finally, publish this report.

• The first step is to convert the file "lcov.dat" obtained during the execution of the tests in "Cobertura.xml" format. This conversion is done with a Python script that is available here. Add this script into the repository, and execute it during the build with a Command Line task.
  

Example:

$(Build.SourcesDirectory)\CountdownWidget\lcov_cobertura.py $(Build.SourcesDirectory)\CountdownWidget\CountdownWidget\lcov.dat --output $(Build.BinariesDirectory)\Cobertura.xml

• It is necessary to generate a report in html format from the "Cobertura.xml" using the tool ReportGenerator which is obtained in a package Nuget (included in our project), available here.
  

Examples:

$(Build.SourcesDirectory)/CountdownWidget/packages/ReportGenerator.2.5.1/tools/ReportGenerator.exe

-reports:$(Build.BinariesDirectory)\Cobertura.xml -targetdir:$(Build.BinariesDirectory)\reports -reporttypes:Html;Xml;XmlSummary

• The last step is to publish the report generated by ReportGenerator by using the Publish Code coverage result task.
  

Examples:

$(Build.BinariesDirectory)/Cobertura.xml

$(Build.BinariesDirectory)/reports

Once the build is running, you get the report in the Code Coverage tab.

1. Add Jasmine to your project
2. Add and validate unit tests
3. Add and configure Test Assemblies task to your CI build
4. Configure Chutzpah configuration to enable code coverage
5. Publish code coverage results

• Anonymous
  February 25, 2017
  There are easier ways to publish the code coverage report, since Chutzpah can also output in JaCoCo format and an HTML report, using the Transforms settings (it can output multiple formats at once). See here https://medium.com/nov-rsis-development/enabling-javascript-typescript-code-coverage-in-a-vsts-tfs-build-36fe41d40f97#.qe6ly49kq
  • Anonymous
    February 28, 2017
    Chutzpa provide xml summary JaCoCo code coverage but doesn't provide the Html report Jacoco or Cobertura format for the Code coverage tab in the build, so I need to use Report Generator for tranform xml summary to html compatible report
• Anonymous
  March 09, 2018
  Any hints on using this with typescript (and webpack)? I tried addingimport {} from 'jasmine';to the top of your example spec for adding two numbers, but when I do npm run test it says 'define is not defined':C:\git\myproject>npm test> myproject@0.0.1 test C:\git\myproject> jasmine-ts "src/spec/**/*.spec.ts"C:\git\myproject\src\spec\test.spec.ts:1import {} from 'jasmine'; ^ReferenceError: define is not defined at Object. (C:\git\vsts-team-calendar\src\spec\test.spec.ts:1:63) at Module._compile (module.js:635:30) at Module.m._compile (C:\git\myproject\node_modules\ts-node\src\index.ts:392:23) at Module._extensions..js (module.js:646:10) at Object.require.extensions.(anonymous function) [as .ts] (C:\git\myproject\node_modules\ts-node\src\index.ts:395:12) at Module.load (module.js:554:32) at tryModuleLoad (module.js:497:12) at Function.Module._load (module.js:489:3) at Module.require (module.js:579:17) at require (internal/module.js:11:18)npm ERR! Test failed. See above for more details.C:\git\myproject>I'm also using webpack, not sure if I have to do something with that?
• Anonymous
  April 27, 2018
  Under the heading "In Visual Studio Code", you say: "You can use the grunt-contrib-jasmine grunt already existing in the template, and in the "gruntfile.js" file have the jasmine task."What project template? The only project template I see is the Visual Studio template you mentioned, but this isn't Visual Studio, it's VS Code.
  • Anonymous
    April 29, 2018
    Hi,The project template that I memtion is the VS project template for VSTS extension provided by Josh Garverick.In the VS code section I expose how run the js tests from VS Code, but you need to install the template in VS.However, I suggest to you also read this article https://blogs.msdn.microsoft.com/visualstudioalmrangers/2017/03/16/generate-team-services-extension-starter-kit-with-the-yeoman-generator-vsts-ext/ which is more recent about templates for VSTS extensions
    • Anonymous
      May 01, 2018
      Thanks, any guidance on using this with typescript? Jasmine keeps saying "define is not defined".
      • Anonymous
        May 01, 2018
        Hi,You can see a sample with TypeScript tests on the Countdown Widget , here https://github.com/ALM-Rangers/Countdown-Widget-Extension/tree/master/CountdownWidget/CountdownWidget/tests
        • Anonymous
          May 08, 2018
          (The content was deleted per user request)
        • Anonymous
          May 08, 2018
          The comment has been removed
        • Anonymous
          December 05, 2018
          has same issue with testing with VSS sdk imported, any idea what to do?