Use SonarQube quality gates to control your Visual Studio Team Services builds

Bogdan Gavril

Note: a more recent documentation is available from Analyzing with SonarQube Extension for VSTS/TFS

In Visual Studio Team Services and Team Foundation Server you can cause a build to fail if the code does not meet the conditions imposed by a SonarQube quality gate. This post describes how to configure this when using SonarQube 5.3 and earlier, and discusses other related issues:

• Failing the build on quality gate violations with SonarQube 5.3 or later
• Failing the build when using earlier versions of SonarQube
• SonarQube analysis and “pull request” builds
• The default quality gate and changes to the project version

Failing the build on quality gate violations with 5.3 or later

The build task “SonarQube for MSBuild – Begin Analysis” in Team Services and TFS has a new option: break the build when the quality gate associated with the SonarQube project is not met. You need to use a SonarQube server version 5.3 or later for this. This was possible with older versions of SonarQube only by using the “Build Breaker” plug-in. In the new “SonarQube for MSBuild – Begin Analysis” build task, you can set the Fail the build on quality gate failure checkbox when you are using SonarQube 5.3 or later.

Setting this option causes the “SonarQube for MSBuild – End Analysis” task step to fail if the quality gate associated with the project fails; which, in turn, causes the entire build to fail and generate a SonarQube quality gate error message.

Note that, even though version 5.2 and later of SonarQube allowed some processing to take place by on build agent in order to improve performance, the quality gate check is part of the server-side processing. Therefore, you will see a minor effect on overall build performance when you use this option because the task has to wait for server-side processing to complete so that the quality gate can be evaluated.

Failing the build when using earlier versions of SonarQube

Depending on the version of SonarQube you are using, consider the following options for causing the build to break when the quality gate fails:

• SonarQube 4.x: you can install and use the SonarQube “Build Breaker” plug-in.
• SonarQube 5.2: you should upgrade to SonarQube version 5.3 or later. Architectural changes in version 5.2 mean that the “Build Breaker” plug-in cannot be used.

Ideally, you should upgrade to SonarQube 5.3 or later.

SonarQube analysis and “pull request” builds

Team Services and TFS can be configured to initiate a build whenever a pull request (PR) is submitted. However, we do not recommend you perform a SonarQube analysis in a PR build because the SonarQube task will upload analysis results to SonarQube for code that has not been (and may never be) checked in.

However, you can use build break option as part of a continuous integration (CI) build that is initiated in response to changes in your branch, as well as for manually triggered and scheduled run builds.

Note: We are currently working on a separate feature that will integrate code analysis into PR builds without the undesirable side-effects.

The default quality gate and changes to the project version

SonarQube quality gates may use baselines related to a specific project version. For example, the default quality gate in SonarQube 5.3 states that the project should:

• Have a code coverage on new code greater than 80%
• Have no new blocker or critical issues on new code
• Have a Tech Debt ratio on new code greater than 5

There are all relative to the previous version of the project as defined in the build task settings:

Changing the product version number will create a new baseline to compare with. If done too soon, it will prevent the proper operation of the quality gate and consequent build failure – which may result in lack of control of your overall code quality and technical debt. Typically, you should avoid changing the project version more often than is necessary.

Bogdan Gavril

Follow