Set up a CI/CD pipeline with unit testing and code coverage for your Team Services extension


We receive a notification that there’s a pull request to review for our Team Services extensibility project. We discuss the proposed changes as a team, eventually merge the new feature(s) into the master branch, and close the pull request. Done … right?

Not quite. The merge triggers our automated CI/CD pipeline, which consistently builds, tests and delivers our package to the Visual Studio marketplace.


image_thumb2


This post should be a valuable reference, if you’re interested in how we configured our Countdown Widget pipeline and how the team included testing and code coverage. Let’s walk through the configuration of our pipeline.



image_thumb8Build

We start with the continuous integration build, made up of thirteen steps.

The magic around unit testing and code coverage happens during steps 8-11. See Testing a Team Services extension and tracking code coverage for more details.

SNAGHTMLe706a59

image_thumb6


# BUILD STEP BUILD SNIPPET WHAT’S HAPPENING?
1 Whitesource SNAGHTMLf419f3b Scan and detect security vulnerabilities, problematic OSS licenses and quality issues. See Manage your open source usage and security in your pipeline for details. You’ll notice that we’re still experimenting with this task and its location in the build. Here we scan before the build, while in Set up a CI/CD pipeline for your Team Services extension we scan after the build.
Watch our Whitesource tag for updates on our findings.
       
2 Security “stuff” image We’re experimenting with a collection of internal security tools to automate tasks like checking for sensitive term violations as part of the pipeline.
       
3 NuGet restore SNAGHTMLf4d445f Restore NuGet packages as configured in the CountdownWidget.sln solution.
       
4 npm install SNAGHTMLf519059

SNAGHTMLf53ab4a
Install the NPM package dependencies as defined in package.json.
       
5 grunt prepare for build SNAGHTML101ae211

SNAGHTML10224ee3
Perform the copy:main (copy) custom task defined in gruntfile.js copying seven files.
       
6 Replace tokens SNAGHTMLf3f457c

SNAGHTMLf401bc8

We replace the INSTRUMENTATION key in the TelemetryClient.ts file with the value of the variable with the same name. Token Regex is “__(\w+)__”, which means that we’re looking for one or more letter, digit or underscore characters, surrounded by a double underscore “__name__”.

Variables

  • INSTRUMENTATIONKEY defines the Application Insights instrumentation key. We’re using a variable to ensure we do not leak the instrumentation key into the OSS repo.
  • system.debug can be set to true for detailed and verbose logging during the build.
       
7 Build solution SNAGHTML102389b4 Build solution as defined in CountdownWidget.sln.
       
8 Test Assemblies SNAGHTML1029cdd5

Run tests with Visual Studio test runner looking for TypeScript test files named *Spec.ts.

The Chutzpah package, containing Jasmine and its test adapter, were installed to the packages folder as part of step 3 – NuGet restore. 

We are currently running 18 Jasmine based unit tests, as defined in CountdownCalculatorSpec.ts.
       
9 Run python Icov tp cobertura script SNAGHTML10395dc7

Arguments: $(Build.SourcesDirectory)\CountdownWidget\lcov_cobertura.py  $(Build.SourcesDirectory)\CountdownWidget\CountdownWidget\lcov.dat --output $(Build.BinariesDirectory)\Cobertura.xml
Convert the file lcov.dat line coverage output to Cobertura-compatible XML format using the lcov_cobertura.py python script.
       
10 Run Report Generator SNAGHTML103feda6

Tool: $(Build.SourcesDirectory)/
CountdownWidget/packages/
ReportGenerator.2.5.1/tools/
ReportGenerator.exe

Arguments: -reports:$(Build.BinariesDirectory)\Cobertura.xml -targetdir:$(Build.BinariesDirectory)\reports -reporttypes:Html;Xml;XmlSummary
Convert the Cobertura.xml file to HTML format, using the ReportGenerator we installed to the packages folder as part of step 3 – NuGet restore. 
       
11 Publish Code Coverage Results SNAGHTML1043bc42
TEST RESULTS:
image
CODE COVERAGE DETAILS:
SNAGHTML10471020
Publish the code coverage results from the formatted Cobertura.xml file.
       
12 Package Extension SNAGHTMLf3e0154

We package the extension using the VSTS Developer Tools Build Tasks extension and generate the output.vsix artifact.

Extension ID is appended with Alpha to differentiate it from deployed DEV, BETA, and PROD packages.

       
13 Publish Artifact SNAGHTMLf3e82e8

We publish the artifact, output.vsix, to $(Extension.OutputPath) to be picked up by the release.



Release

Next we continue with continuous delivery (CD), made up of three environments. With the exception of configuration values, this part of the pipeline is identical (consistent) to the one we covered in up Set up a CI/CD pipeline for your Team Services extension.

Continuous Delivery is the ability to use the output from the CI to build and deploy the new known good build to one or more environments automatically (bit.ly/28PWsfk). There is a subtle difference between Continuous Delivery and Continuous Deployment. The latter is to a single environment. A small team might only implement Continuous Deployment because each change goes directly to production. Continuous Delivery is moving code through several environments, ultimately ending up in production, which may include automated UI, load and performance tests and approvals along the way. – Extract from DevOps - Applying DevOps to a Software Development Project

image_thumb11DEV

The environment with one task is triggered after a successful release creation. image_thumb43 Both the pre-deployment and post-deployment approvals are automatic, ensuring that the development team is always working with the latest version. There’s no manual intervention.

SNAGHTML21af0038_thumb6[6]

image_thumb9


# RELEASE STEP BUILD SNIPPET WHAT’S HAPPENING?
1 Publish Extension SNAGHTMLe66da88

We publish the extension to the alm-rangers publisher, using the VSTS Developer Tools Build Tasks extension. 

  • The published private extension is shared with our BETA sandbox-one and used for exploratory and user acceptance testing.
  • The extension ID is suffixed with BETA to make it simple to identify this extension.



image_thumb12BETA

The environment with only one task is triggered after a successful deployment to the DEV environment. image_thumb42 The BETA approvals mandate that one of the specified users approve the release, to ensure that a high quality bar can be met. Users typically include the project lead and program manager.

SNAGHTML21af0038_thumb6

image_thumb12[1]


# RELEASE STEP BUILD SNIPPET WHAT’S HAPPENING?
1 Publish Extension SNAGHTMLe6789a4

We publish the extension to the alm-rangers publisher, using the VSTS Developer Tools Build Tasks extension. 

  • The published private extension is shared with our BETA sandbox-one and used for exploratory and user acceptance testing.
  • The extension ID is suffixed with BETA to make it simple to identify this extension.

image_thumb16PROD

The environment with one task is triggered after a successful deployment to the BETA environment. image_thumb41 The PROD approvals mandate that all of the specified users approve the release, to validate that the aka.ms/vsarDoD (definition of done) is met. Users typically include the project lead, program manager, and product owner.

SNAGHTML21af0038_thumb6[9]

image_thumb21


# BUILD STEP BUILD SNIPPET WHAT’S HAPPENING?
1 Publish Extension SNAGHTMLe6816f0

We publish the extension to the ms-devlabs publisher, using the VSTS Developer Tools Build Tasks extension.  

  • The published extension is public and available on the marketplace.



That’s it!

Now that you have an understanding of how our pipeline works, you should explore how you can configure your continuous integration and delivery pipeline by #RubDevOpsOnIt.

Other pipelines examples

image16272image62image112image5[1]

Reference Information

DevOps - Applying DevOps to a Software Development Project, lcov-to-cobertura-xml, Set up a CI/CD pipeline for your Team Services extension, Testing a Team Services extension and tracking code coverage, VSTS Developer Tools Build Tasks

Comments (0)

Skip to main content