How to check that tests cover a commit with new code? How to control that the tests will be written? What type of coverage metric to choose for analysis?

You can use simple tools, to set up Build Validation, the check will start with each Pull Request in the branch selected for validation and include the Code Coverage check steps, stopping the build if the percentage of coverage has decreased. This approach is called “Quality Gate”.

A Quality Gate is a mandatory measure or metric built into the Pipeline that the software must meet before moving on to the next step. This measure enforces certain rules, metrics, or practices that code must follow to prevent low-quality code from entering the software being developed.

Quality Gate can be easily configured in Azure Pipelines. The steps of assembly and coverage analysis discussed in the article are universal and can be used in any CI/CD system, such as Jenkins, TeamCity, etc.

To easily test the solution, you will need:

tutorial project: https://github.com/devops-stack/eShopOnContainers

create an account in Visual Studio or log in with a GitHub account: https://azure.microsoft.com/en-us/products/devops/pipelines

Project preparation and launch

For clarity, it is best to start with a practical example. In a free subscription, you can clone a repository and create your pipeline, the creation and launch process is simple and takes 3 minutes. In the free Azure DevOps subscription (VisualStudio), there is an available build agent, so you can quickly see everything yourself:

Register or log in with a GitHub account (the registration process is as easy as possible): https://azure.microsoft.com/en-us/products/devops/pipelines and click the “Start Free” button.

Then click “+ New Project”, enter the name of the project in the “Project Name” field, and click “Create”. After creation, go to the “Repos” section, in the “Import a repository” section, click “Import”, copy the link https://github.com/devops-stack/eShopOnContainers into the “Clone URL” field, and click “Import”.

After cloning, go to the “Pipelines” section, click the “Create Pipeline” button, select “Azure Repos Git”, then select the newly created repository and click “Existing Azure Pipelines YAML file”. Select the “main” branch, copy the “pipeline path “src/Services/Catalog/azure-pipelines.yml”, then “Continue” and click “Run”. Your build is ready!

An example of a PROD-level project

The article uses a project developed by the Microsoft team, it is a ready-to-use online store developed on.NetCore, using microservice architecture and meeting design and development standards. Unit Tests and Functional Tests have also been added, so the store is convenient to use for demonstrations and practical examples (I tried to tell you about this development in more detail). The project is conveniently deployed locally, in Kubernetes, Docker Compose, or Azure Kubernetes Service. The repository also includes several books that fully cover the development and operation cycle, which are very convenient for studying technology from different perspectives such as QA, Development, and DevOps. The code and tests of the “Catalog” service of this online store will be used in the article.

To simplify the example and clarity, only one of the services of the project will be tested, and in the pipeline, direct paths to the project files are used, while in real environments, variables are more often used. To analyze the coverage of a repository with a large number of services, the process is slightly complicated. In the second article, a universal template for different services using variables will be considered.

Pipeline steps with Code Coverage and Build Quality Checks

Tasks #1 and #2: Restore and Build

The first two tasks update dependencies and assemble the service (the main branch is used throughout the article):

Task #3 Starting tests and forming a data collection

To collect data on the volume of the tested code, we will use the Coverlet collector with the help of the –collect “XPlat Code Coverage” option. This key is used for Linux environments. It’s a cross-platform option based on the .NET CLI that’s great for building systems where MSBuild isn’t available. In the Windows environment, you can use the –collect “Code Coverage” parameter. Cobertura will be used to calculate the percentage of code that the tests cover.

for the successful execution of this step, a dependency must be added to the csproj file:

<PackageReference Include=”coverlet.collector” Version=”6.0.0″ />

Task #4 Publication of collected statistics

The step “Publish Code Coverage Results” will be needed for the final stage, to prepare the artifact in Cobertura format and mark it in the required location.

After completing steps 3 and 4, the “Tests” and “Code Coverage” tabs will become available.

Information about the execution of tests and detailed “Code Coverage” statistics are displayed on the tabs, respectively: 

Task #5 Setting up Build Quality Checks

The “Build Quality Checks” step allows you to add a “Quality Gate” to the pipeline. Like the previous stages, this step is quite simple, but it is the “icing on the cake”, we will use the policy of the Azure DevOps branch to configure the stop of the build if the coverage threshold has decreased. This will protect the project from reducing the level of code coverage by tests. “Build Quality Checks” at the first launch analyze the code coverage and form a Baseline – the basic value from which the decrease in code coverage will be calculated, this is simply the percentage of coverage of the previous successful build. You can also set the desired coverage percentage.

An example with a fixed coverage percentage set manually:

Code Coverage and Branch Coverage 

The two most popular coverage metrics are code coverage and branch coverage. In the final step “Build Quality Checks”, analysis with the Code Coverage metric is selected by default. It’s easier to start implementing the metric by monitoring the decrease in the percentage of coverage compared with the previous build and developing a suitable percentage based on the experience of developing project requirements. I repeat, if you have just started working with metrics, it is better to start simple, analyze build by build how exactly the percentage of code coverage for your project is formed, and not make strict requirements for coverage when.

Branch coverage provides deeper analysis than code coverage. Instead of using the number of lines of code, this metric focuses on structures such as if and switch statements and makes test development a little more difficult.

Visual Studio Test evaluates how many lines of code, as well as how many blocks of code, were covered by tests. Other tools, such as Cobertura, use other types of coverage, such as Branch Coverage, which is similar to Block Coverage.

Quality Gate: Setting up an Azure DevOps branch policy

Let’s do a simple setup for Quality Gate in Azure Pipelines:

In Azure Pipelines, go to “Repos” -> “Branches”, hover over the “main” branch and on the far right, select the three-dot button that appears and then “Branch Policies”.

In the “Build Validation” section, click “+”, check that the pipeline that we created at the beginning of the article is selected in the field, and click “save”.

Let’s check the prepared solution:

Create a new branch from the main, in the “repos” > “Branches” section, click “New Branch”, give the branch a name, and make sure that the “main” branch is selected in the “Based on” field.

In the “Branches” section, go to the created branch (use the search), open the file “/src/Services/Catalog/Catalog.UnitTests/Application/CatalogControllerTest.cs” and delete all the code in the “Get_catalog_items_success()” method, the method itself and two Leave the {} brackets (see screenshot below).

Then click “Commit” > “Commit” and the “Create a pull request” button that appears. Fill in the “Title” field and click “Create”

Now the main branch is protected by the pipeline that we developed. When you try to make a PR to a branch, the pipeline will launch and if the percentage of code coverage decreases, it will stop with a message that the percentage of code coverage has decreased relative to the last review. And the configured branch policy will not allow the pull request to be completed.

The screenshot above shows that verification is required when PR is made to a branch with the “Build Validation” policy enabled. A message appeared about reduced code coverage and the build stopped. By clicking on the name of the pipeline, you can view the details.

The execution of a Pull Request does not have to be “stopped”. If you are just forming requirements for a testing approach, it is convenient to set up an optional check and not stop the build. Also in Azure DevOps it is possible to grant rights to change the branch policy to certain people. Thus, when the testing approach, metrics, and percentage of coverage are developed, if there is a need, you can strictly regulate the requirements for test coverage.

The next article will add an important step in setting up a template pipeline for a repository with a large number of services. This is necessary if the services of your platform or web application are developed by different teams. Then the Quality Gate pipeline should not block the development of other services.