Performance Tuning

Although pitest can analyse mutants orders of magnitude faster than earlier systems, for some code bases this can still seem very slow.

These performance concerns are largely eliminated when arcmutate’s git integration is used, and we recommend that mutation testing is applied this when wherever possible.

However, some teams require metrics that are calculated across an entire codebase. This cannot be achieved using git integration, and usually requires a complete analysis run.

Arcmutate’s history feature can reduce the cost of re-calculating whole-of-codebase metrics, and can it be combined with git integration to further reduce the computational cost.

In tandem with these approaches, the following techniques can also be used to reduce analysis times.

Use More Threads

By default, pitest runs in a single thread regardless of the capabilities of the environment. This ensures that builds are reproducible across machines, but under-utilises most hardware.

A higher fixed number of threads can be used, or the ‘+auto_threads` feature can be activated. This attempts to pick the optimum number of threads based on the available cores.

Use Test Acceleration

Some codebases using JUnit 5 will benefit from the arcmutate accelerator plugin. It is not able to speed up all codebases, but for some the improvement is significant.

Exclude Slow Tests

Pitest challenges each mutant with tests until one of them kills the mutant, or there are no tests remaining that exercise the mutated code. The fastest tests are run first, slower ones are only run if the faster tests did not kill the mutant.

If the codebase contains slow running tests that exercise large amounts of code, they may have a disproportionate effect on analysis times as they will be run for each surviving mutant.

Excluding these tests from the analysis may be a pragmatic choice for some codebases.

Use Extreme Mutation Testing

Pitest’s fine-grained mutation operators provide actionable feedback for developers. If your goal is instead to produce metrics for tracking test quality, the coarser extreme operators may be appropriate. These will produce fewer mutants, reducing the computational cost.

Use a Reduced Operator Set

Alternately, a smaller selection of the fine-grained set could be used. Teams have reported useful results using just the following two operators

• CONDITIONALS_BOUNDARY
• VOID_METHODS

Exclude Code

A simple strategy to reduce analysis times is to reduce the scope of where mutation testing is applied. Pitest and arcmutate provide

Two common approaches are to target only mission critical code, or to exclude integration code.

Mission Critical Code Only

Only the code considered most critical is targetted.

Exclude Integration Code

Code with a pure integration concern (e.g. DAOs or http clients) is excluded. This code is often tested only via slow running integration tests, so is the most expensive code to mutate.

If concerns have been well separated within the codebase it will contain no business logic, so it is also less likely to yield interesting results.