Set up a CI/CD pipeline with unit testing and code coverage for your VSTS extension

We receive a notification that there’s a pull request to review for our Team Services extensibility project. We discuss the proposed changes as a team, eventually merge the new feature(s) into the master branch, and close the pull request. Done … right?

Not quite. The merge triggers our automated CI/CD pipeline, which consistently builds, tests and delivers our package to the Visual Studio marketplace.

image_thumb2

This post should be a valuable reference, if you’re interested in how we configured our Countdown Widget pipeline and how the team included testing and code coverage. Let’s walk through the configuration of our pipeline.

MODIFIED 2017.08.14 – Update to align with latest codebase from https://github.com/ALM-Rangers/Countdown-Widget-Extension. If you remember the previous pipeline, you’ll notice that the pipeline has been simplified by code generated by the the fastest path to a new VSTS extension generator.


image_thumb8Build

We start with the continuous integration build, made up of thirteen steps.

The magic around unit testing and code coverage happens during steps 8-11. See Testing a Team Services extension and tracking code coverage for more details.

SNAGHTML5c4789

image_thumb6

# BUILD STEP BUILD SNIPPET WHAT’S HAPPENING?
1 Install npm dependencies SNAGHTML5da999 Install all modules listed as dependencies in package.json.
       
2 npm custom SNAGHTML5ec70f Run a custom npm task (install –g whitesource) to install the Whitesource npm plugin.
       
3 Replace tokens in Whitesource config SNAGHTML5f87ee

We replace the WhiteSourceAPIKey in the whitesource.config.json file with the value of the variable with the same name. Token Regex is “__(\w+)__”, which means that we’re looking for one or more letter, digit or underscore characters, surrounded by a double underscore “__name__”.

       
4 Replace tokens for TelemetryClient SNAGHTMLf519059SNAGHTMLf53ab4a

We replace the INSTRUMENTATION key in the TelemetryClientSettings.ts file.

Variables

  • INSTRUMENTATIONKEY defines the Application Insights instrumentation key. We’re using a variable to ensure we do not leak the instrumentation key into the OSS repo.
  • system.debug can be set to true for detailed and verbose logging during the build.
       
5 Run whitesource SNAGHTML6203bc Run the Whitesource tool to scan and detect security vulnerabilities, problematic OSS licenses and quality issues. See Manage your open source usage and security in your pipeline for details.
       
6 npm run build SNAGHTML631b56 Run the package build script, as defined under “scripts” in package.json
       
7 nom run tests SNAGHTML674aa0 Run the package test script, as defined under “scripts” in package.json.
     
8 publish test results SNAGHTML690d11 Publish the test results, located in the ./reportTests/ folder.
   
9 Publish Code Coverage Results

SNAGHTML6d0368

RESULTSSNAGHTML73e61a

Publish the code coverage results from the formatted cobertura-coverage.xml file.
       
10 Package Extension SNAGHTMLf3e0154

We package the extension using the VSTS Developer Tools Build Tasks extension and generate the output.vsix artifact.

Extension ID is appended with Alpha to differentiate it from deployed DEV, BETA, and PROD packages.

       
11 Publish Artifact SNAGHTMLf3e82e8

We publish the artifact, output.vsix, to $(Extension.OutputPath) to be picked up by the release.


Release

Next we continue with continuous delivery (CD), made up of three environments. With the exception of configuration values, this part of the pipeline is identical (consistent) to the one we covered in up Set up a CI/CD pipeline for your Team Services extension.

Continuous Delivery is the ability to use the output from the CI to build and deploy the new known good build to one or more environments automatically ( bit.ly/28PWsfk ). There is a subtle difference between Continuous Delivery and Continuous Deployment. The latter is to a single environment. A small team might only implement Continuous Deployment because each change goes directly to production. Continuous Delivery is moving code through several environments, ultimately ending up in production, which may include automated UI, load and performance tests and approvals along the way. – Extract from DevOps - Applying DevOps to a Software Development Project

image_thumb11DEV

The environment with one task is triggered after a successful release creation. image_thumb43 Both the pre-deployment and post-deployment approvals are automatic, ensuring that the development team is always working with the latest version. There’s no manual intervention.

SNAGHTML21af0038_thumb6[6]

image_thumb9

# RELEASE STEP BUILD SNIPPET WHAT’S HAPPENING?
1 Publish Extension SNAGHTMLe66da88

We publish the extension to the alm-rangers publisher, using the VSTS Developer Tools Build Tasks extension. 

  • The published private extension is shared with our BETA sandbox-one and used for exploratory and user acceptance testing.
  • The extension ID is suffixed with BETA to make it simple to identify this extension.

image_thumb12BETA

The environment with only one task is triggered after a successful deployment to the DEV environment. image_thumb42 The BETA approvals mandate that one of the specified users approve the release, to ensure that a high quality bar can be met. Users typically include the project lead and program manager.

SNAGHTML21af0038_thumb6

image_thumb12[1]

# RELEASE STEP BUILD SNIPPET WHAT’S HAPPENING?
1 Publish Extension SNAGHTMLe6789a4

We publish the extension to the alm-rangers publisher, using the VSTS Developer Tools Build Tasks extension. 

  • The published private extension is shared with our BETA sandbox-one and used for exploratory and user acceptance testing.
  • The extension ID is suffixed with BETA to make it simple to identify this extension.

image_thumb16PROD

The environment with one task is triggered after a successful deployment to the BETA environment. image_thumb41 The PROD approvals mandate that all of the specified users approve the release, to validate that the aka.ms/vsarDoD (definition of done) is met. Users typically include the project lead, program manager, and product owner.

SNAGHTML21af0038_thumb6[9]

image_thumb21

# BUILD STEP BUILD SNIPPET WHAT’S HAPPENING?
1 Publish Extension SNAGHTMLe6816f0

We publish the extension to the ms-devlabs publisher, using the VSTS Developer Tools Build Tasks extension.  

  • The published extension is public and available on the marketplace.

That’s it!

Now that you have an understanding of how our pipeline works, you should explore how you can configure your continuous integration and delivery pipeline by #RubDevOpsOnIt.

Other pipelines examples

image16272image62image112image5[1]

Reference Information

DevOps - Applying DevOps to a Software Development Project, lcov-to-cobertura-xml, Set up a CI/CD pipeline for your Team Services extension, Testing a Team Services extension and tracking code coverage, VSTS Developer Tools Build Tasks