Incremental Analysis and History Files

Requires pitest 1.15.0 or above.

Provided by the arcmutate-base plugin.

Background

Pitest supports incremental analysis, where the results of previous runs against a codebase are used to optimise the performance of the current analysis.

Incremental analysis can drastically reduce analysis time, with the caveat that the optimisations applied could introduce some inaccuracies in the results compared to running a full non incremental analysis. These limitations are discussed on the pitest site.

Arcmutate provides an improved incremental analysis implementation which can be activated with the feature string +arcmutate_history. The feature accepts an optional parameter run_tests, which is discussed in the sections below.

The Arcmutate incremental analyser can achiever faster analysis times than the default pitest analyser whilst providing several other practical improvements.

Human Readable Files

Arcmutate generates human readable plain text history files. The output is also more stable than the opaque files produces by pitest which can vary between runs despite apparently identical inputs.

When the run_tests parameter is set to false, Arcmutate guarantees that history files will be unaltered when based on the same inputs. When the run_tests is set to true, output may differ slightly in some code bases due to differences in the order that tests trigger static initialisation.

Note that incremental analysis analysis relies on comparing hashes of the binary class files to determine what has changed between runs. The classes produced by different versions of javac, or by javac on different platforms, may differ. If history files are shared between machines care must be taken to ensure that each environment produces identical bytecode.

Partial Run Consolidation

Information about previous mutants is maintained, even if those mutants are excluded from the current analysis. This enables several patterns of working not possible with the default pitest analyser.

For example, if a full pitest run is prohibitively expensive for a codebase, a file can be built up gradually via repeated partial runs that target only small sections of the codebase. Or the history file can be built up in combination with Arcmutate’s git integration.

To prevent file growth, the corresponding mutants are removed from the history file when classes are deleted from the classpath.

Test Skipping

Normally, incremental analysis requires the entire test suite to be run to determine which tests cover which mutants. For some codebases this can represent a significant amount of processing, sometimes more than the mutation analysis stage if pitest is only analysing a slice of the codebase (e.g. with git integration).

The arcmutate_history feature accepts a parameter named run_tests. By default, this is set to true, but it can be explicitly set to false

+arcmutate_history(run_tests[false])

When running in this mode, Arcmutate will attempt to use historic information to reduce the number of tests coverage is gathered for. In addition, previously ineffective tests will not be used to challenge surviving mutants, further reducing analysis times.

This mode can be used to create a very fast feedback loop when performing incremental analysis against local changes, or to reduce analysis times when running on CI.

When the flag is set to false, pitest does not have access to full line coverage information. For this reason, the line coverage sections of the html report are set to ‘n/a’ so as not to present a misleading picture.

It is possible that using this mode will introduce inaccuracies into the analysis. The decision to rerun tests is based on whether the hashes of the tests classes (and their parent hierarchy) have changed. If changes to other files (e.g CSV or XML resources) can alter the result of a test, Arcmutate will not be able to detect this, and a stale historic result may be recycled.

History File Compatibility

The Arcmutate analyser cannot read history files produced by the pitest analyser, and vice versa. Care should be taken not to mix the file types when upgrading.

Installation

To you use the plugin you must first acquire a licence.

The licence file must be named arcmutate-licence.txt and placed at the root of the project.

The plugin itself should be placed on the classpath of the pitest build integration plugin.

E.g for maven

<plugin>
  <groupId>org.pitest</groupId>
  <artifactId>pitest-maven</artifactId>
  <version>1.16.0</version>
    <dependencies>
      <dependency>
        <groupId>com.arcmutate</groupId>
        <artifactId>base</artifactId>
        <version>1.3.0</version>
      </dependency>
    </dependencies>
</plugin>

Or for gradle

dependencies {
  pitest 'com.arcmutate:base:1.3.0'
}

Configuration

In addition to activating the feature, pitest must also be supplied with the path of an input and output history file to use. If these paths are not supplied, incremental analysis will not be active.

Most often, the input and output file will be the same.

e.g. for maven, if pitest has been bound to a profile named pitest, an incremental run could be performed with

 mvn -Ppitest -DhistoryInputFile=src/test/history.txt -DhistoryOutputFile=src/test/history.txt -Dfeatures="+arcmutate_hisory(run_tests[false])"

Note that for gradle, the input and output file parameters are named ‘historyInputLocation’ and historyOutputLocation.

A file is created for each module of a multimodule build. In the maven example given above the file would be created within the root of the src/test directory of each module. History files can be checked into version control, but teams should be aware that files can grow to be large and will change in multiple locations each time the project source code is modified.