Link Search Menu Expand Document
arcmutate

Adding Pitest To a Maven Project

Pitest provides a maven plugin to run analysis. The CDG plugins require pitest-maven 1.15.0 or above. The latest version is shown below.

Maven Central

We recommend creating a profile to run pitest. This allows pitest to be bound a phase so that it is always run against the latest version of the project’s bytecode, without forcing it to be run during a normal build.

You may wish to place the bulk of the configuration within the pluginManagement section, placing only the execution binding within the profile, but the example below shows the simplest arrangement where all configuration is placed within the profile.

<profile>
  <id>pitest</id>
  <build>
    <plugins>
      <plugin>
        <groupId>org.pitest</groupId>
        <artifactId>pitest-maven</artifactId>
        <version>1.16.0</version>
        <configuration>
            <threads>4</threads>
            <timestampedReports>false</timestampedReports>
        </configuration>
          <executions>
            <execution>
                <id>pitest</id>
                <phase>test</phase>
                <goals>
                    <goal>mutationCoverage</goal>
                </goals>
            </execution>
        </executions>
      </plugin>
  </plugins>
  </build>
</profile>

Pitest can then be run from the command line with

mvn -Ppitest test

The tests will also be run by the surefire plugin. To speedup feedback you may wish to disable the surefire plugin when the pitest profile is active, or instead bind pitest to the test-compile phase. Mutation analysis can then be run with

mvn -Ppitest test-compile

JUnit 5

Pitest works with JUnit 4 out of the box. To use JUnit 5, the pitest junit 5 plugin must be added to the build.

Maven Central

This should be added to the classpath of the pitest-maven plugin, not the classpath of the project.

<profile>
  <id>pitest</id>
  <build>
    <plugins>
      <plugin>
        <groupId>org.pitest</groupId>
        <artifactId>pitest-maven</artifactId>
        <version>1.16.0</version>
        <dependencies>
          <dependency>
            <groupId>org.pitest</groupId>
            <artifactId>pitest-junit5-plugin</artifactId>
            <version>1.1.1</version>
        </dependency>
        <configuration>
            <threads>4</threads>
            <timestampedReports>false</timestampedReports>
        </configuration>
          <executions>
            <execution>
                <id>pitest</id>
                <phase>test</phase>
                <goals>
                    <goal>mutationCoverage</goal>
                </goals>
            </execution>
        </executions>
      </plugin>
  </plugins>
  </build>
</profile>

First Run

Depending on the details of your codebase, running a complete analysis may take anything from a few seconds to a few hours.

For the first run is a good idea to do a reduced analysis to speed up the feedback cycle while flushing out any problems.

mvn -Ppitest -Dfeatures="+CLASSLIMIT" test

Will run all the tests, but only create a single mutant for each class.

This will speed the analysis up considerably in most cases, but cannot be used to estimate the overall analysis time. There is a fixed cost to analysing each class, so additional mutants in the same class will be far less expensive.

Common Issues

Hopefully everything ran without issue, but if you were unlucky you might have got an error like the one below

Execution of goal org.pitest:pitest-maven:1.16.0:mutationCoverage failed: 
32 tests did not pass without mutation when calculating line coverage. 
Mutation testing requires a green suite

Followed by a list of failing tests.

The most common causes for this error are :-

  • A configuration value that is set for surefire, but not for pitest
  • A test order dependency
  • An issue with a particular test or tests

Surefire Config

Pitest attempts to parse the surefire configuration and create an equivalent configuration for pitest.

This process is not perfect however. If tests are failing under pitest, check if they are excluded in the surefire config or if they rely on variables that are set in the surefire configuration. This configuration may need to be manually replicated for pitest.

Test Order Dependencies

A test order dependency is when the results of the tests are different when the order in which they are run is changed.

Test order dependencies can be very difficult to spot, and most teams are convinced they do not have them until they have been found.

Common sources of test order dependencies are

  • mutable static state
  • tests that rely on shared databases
  • tests that modify state

If a test fails under pitest, try updating the pitest configuration so that only that test is run using the targetTests parameter. If the test passes on its own it may be part of an order dependency with other tests in the suite.

Issues With Particular Tests

Occasionally pitest is unable to run a test that uses a particular custom JUnit 4 runner. This might be the case if the test passes when run via surefire, and does not fail when run individually via pitest.

If you believe you have encountered such as case, create a minimal project that reproduces the problem and raise an issue against pitest.

The test can then be excluded during the analysis using the excludedTestClasses parameter.

<plugin>
  <groupId>org.pitest</groupId>
  <artifactId>pitest-maven</artifactId>
  <version>1.16.0</version>
  <configuration>
      <excludedTestClasses>
        <param>your.package.TestToExcludeTest</param>
        <param>supports.glob.wildcards.*.*Bad*Test</param>
      </excludedTestClasses>
  </configuration>
</plugin>