GitHub Integration

Projects with the git plugin configured can be setup to run pitest against pull requests, mutating only the lines of code affected by the changes. This is a very effective way to use mutation testing. Analysis usually takes only seconds, no matter how large the codebase.

The results are displayed directly in the PR. A comment is created each time changes are pushed, showing a summary of the mutation testing results.

example GitHub comment

An annotation is also added to the GitHub diff view for each line of code with surviving mutants.

example GitHub annotation

GitHub integration is easiest using GitHub Actions. Other CI systems are also supported, but require some additional steps in order to authenticate with the GitHub API.

Plugins are provided for maven and gradle

Licence

Before you can use the integration, you must first acquire a licence.

Once you have the licence you should place the file it in the root of your git repo. Do not change the name from arcmutate-licence.txt

Maven Configuration For GitHub Actions

GitHub integration is provided by the pitest-github-maven-plugin

Before configuring the maven plugin, first configure the pitest-git plugin.

The CDG pitest-git plugin and pitest-github-maven-plugin are always released together, and their version numbers should be kept in sync. This can be achieved by creating a property.

<properties>
  <cdg.pitest.version>2.0.0</cdg.pitest.version>
</properties>

The pitest-github-maven-plugin can then be added to the pluginManagement section of your pom (or plugins section if it is a single module project).

<plugin>
    <groupId>com.arcmutate</groupId>
    <artifactId>pitest-github-maven-plugin</artifactId>
    <version>${cdg.pitest.version}</version>
</plugin>

An action can the be created to run pitest against all PRs

name: pitest
on:
  pull_request:

jobs:
  pull-request-ci:
    # Only run on PRs from the repo. PRs from forks will fail due to lack of permissions and
    # must use a two stage process
    if: github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    steps:
      - name: Checkout project
        uses: actions/checkout@v2
        with:
          # important to set a fetch depth of 2. By default the checkout action make no history available
          fetch-depth: 2
      - name: Cache local Maven repository
        uses: actions/cache@v2
        with:
          path: ~/.m2/repository
          key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
          restore-keys: ${{ runner.os }}-maven-
      - name: Setup Java JDK
        uses: actions/setup-java@v1.4.3
        with:
          java-version: 11
      - name: run pitest
        run: mvn -e -B -Ppitest -Dfeatures="+GIT(from[HEAD~1]), +gitci" test
      - name: update pull request
        run: mvn -e -B -DrepoToken=${{ secrets.GITHUB_TOKEN }} pitest-github:github
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

There are some important things to note in this setup.

The fetch-depth of the checkout action has been increased to 2 so git history is available.

The git-plugin has been configured to look for all changes since HEAD~1. Although at first glance it seems that this would mean only the latest commit in a branch was analysed, this is not the case. GitHub places all commits in a PR in a single commit so pitest will analyse the full change.

The +gitci parameter passed to pitest instructs it to produce additional json output that is read by the pitest-github plugin.

If the optional LEVEL parameter is set, the check run status will be set as follows if there are any surviving mutants.

error - check run failure
warning (default) - check run warning
info - check run notice

e.g.

mvn -Ppitest -Dfeatures="+GIT(from[HEAD~1]), +gitci(level[error])" test

See the demo project for a full example of a configured project.

Gradle Configuration for GitHub Actions

Setup with gradle is similar to maven.

The com.arcmutate.github plugin provides GitHub integration.

If the token provides by GitHub actions is placed in an environment variable named GITHUB_TOKEN no additional configuration should be required after the plugin has been added to the project.

This can be seen in action in the example gradle project

Permissions for GitHub Actions

The plugin requires write permissions in order to update the pull request. By default the token provided by GitHub actions provides only read level access. This can be updated in the repository settings, by selecting “Read and write permissions” under Workflow permissions. It may be necessary to change this setting within the Github organization before it can be changed within the repository.

GitHub provides options to set the permissions of the token on a fine grained level.

Untrusted Forks

The previous configuration works when PRs are created from trusted branches within a single repo. If you are working on a project where PRs come from untrusted forks, follow the additional steps described here.

Other CI Systems

Authentication

If you do not wish to use GitHub Actions additional steps are required to allow the pitest-github-maven-plugin to authenticate with GitHub.

First, create a new GitHub App for your organisation. This is more straightforward than it sounds.

On GitHub navigate to Developer settings/Github Apps.

You will need to provide

A unique name (e.g yourorg-pitest)
A homepage URL (e.g. http://yourorg.com)

You do not need to provide a webhook URL, and the webhook checkbox can be unticked.

The app will need the following permissions

Checks - Read & write
Issues - Read & write
Pull Requests - Read & write

GitHub will generate an App ID for your new app, make a note of it as you will need it later.

Once the app has been created, install it for your organisation and give it access to all repositories you wish to mutation test.

It is important that there is only a single installation of the app.

Next, generate and download private key for your app in PKCS#8 format.

GitHub will generate a key for the app in PKCS#1 format

This can be converted to PKCS#8 using openssl

openssl pkcs8 -topk8 -inform PEM -outform PEM -in key-downloaded-from-github.pem -out app-key.pem -nocrypt

The app-key.pem file should be stored on your CI server using the facilities it provides for securing sensitive files. The details of this will be different for each CI sever.

The plugin can now authenticate using the combination of the App ID and secret key.

Other Parameters

The following parameters must be manually set when not using GitHub Actions

githubRepository - The name the git repository
githubUrl - The GitHub instance api URL
githubEventPath - The path the to pull request json payload
appId - The id of the app created for authentication
appKeyPath - The path the the private key of the app

See the maven and gradle plugins for build system dependent details.

Generating `gitci` Json

Before the GitHub plugin is invoked, pitest must generated files in gitci format. This requires access to the PR branch source. It is important that shallow cloning is not used as the pitest plugin requires access to the git history.

In the GitHub Actions environment, analysis is performed against HEAD~1, which will contain all changes within the PR.

mvn -e -B -Ppitest -Dfeatures="+GIT(from[HEAD~1]), +gitci" test
mvn -e -B -DappId=<APP_ID> -DappKeyPath=/path/to/app-key.pem pitest-github:github

This may differ, depending on the details of your CI system. The ref used should represent the state of the repo at the start of the PR branch.