Azure Devops Integration

Projects with the git plugin configured can be setup to run pitest against pull requests on Azure Devops cloud. This is a very effective way to use mutation testing. Analysis usually takes only seconds, no matter how large the codebase.

The results are displayed directly in the PR. A comment is created each time changes are pushed, showing a summary of the mutation testing results.

And annotations are added to the diff view for each line of code with surviving mutants.

Plugins are provided for maven and gradle

Licence

Before you can use the integration, you must first acquire a licence.

Once you have the licence you should place the file it in the root of your git repo. Do not change the name from arcmutate-licence.txt

Maven Configuration

Azure Devops integration is provided by the pitest-azure-maven-plugin

Before configuring the maven plugin, first configure the pitest-git plugin.

The pitest-git plugin and pitest-azure-maven-plugin are always released together, and their version numbers should be kept in sync. This can be achieved by creating a property.

<properties>
  <cdg.pitest.version>2.0.0</cdg.pitest.version>
</properties>

The pitest-azure-maven-plugin can then be added to the pluginManagement section of your pom (or plugins section if it is a single module project).

<plugin>
    <groupId>com.arcmutate</groupId>
    <artifactId>pitest-azure-maven-plugin</artifactId>
    <version>${cdg.pitest.version}</version>
</plugin>

A pipeline can then be created to run pitest. This should be configured within Azure to be run for PRs only.

It is important that shallow fetch is disabled or else git history will not be available for the plugin.

steps:
  - task: Maven@3
    inputs:
      mavenPomFile: 'pom.xml'
      goals: '-Ppitest -DoutputFormats=gitci -Dfeatures=+GIT(from[HEAD~1]) test'
  - task: Maven@3
    env:
      SYSTEM_ACCESSTOKEN: $(System.AccessToken)
    inputs:
      mavenPomFile: 'pom.xml'
      javaHomeOption: 'JDKVersion'
      goals: '-e pitest-azure:azure-devops'

Azure creates a branch containing all PR changes within a single commit. The arcmutate git plugin is therefore configured to look at changes from HEAD~1.

The gitci output format parameter passed to pitest instructs it to produce additional json output that is read by the pitest-azure plugin.

The parameters required by the pitest-azure plugin are automatically resolved from environment variables. Most of these are provided automatically, but the token used to access the Azure api must be explicitly exported.

Azure provides a token under System.AccessToken linked to the Build Service user. By default this user does not have the PullRequestContribute permission required to add comments to the pull request. This permission must be added via the Azure interface, or a different personal access token used.

Gradle Configuration

Setup with gradle is similar to maven. See azure gradle plugin for details.