JUnit 5 Accelerator Plugin

Release 1.1.0 requires pitest 1.15.0 or above.

Background

Mutation analysis times depends on many different factors which make overall execution times difficult to predict, but the execution time of the tests is always an important variable.

Although small improvements in the execution time of individual tests are of little importance when the tests are run only once within a suite, mutation testing may require each test be run thousands of times. Small speed gains can become of practical importance.

The junit5-accelerator plugin improves mutation test performance by reducing the overhead in running tests that use a limited set of JUnit Jupiter features. Tests that use only these features will run more quickly, tests that use features outside this set will be executed as normal.

For some codebases this can have a large impact on performance, e.g analysis time for Commons Lang is reduced from over 55 minutes to under 18.

Eligible tests

Tests that use the following subset of JUnit 5 Jupiter functionality will be accelerated. Any test classes using features/annotations outside of this set will be run as normal. Note that this is an “all or nothing” approach, an unsupported annotation added to one test method means no tests within that class will be accelerated.

• @Test (parameterless methods only)
• @BeforeEach
• @AfterEach
• @DisplayName (annotation is ignored)
• @BeforeAll
• @AfterAll

The following mockito features are also supported

• @ExtendWith(MockitoExtension.class)
• @Mock (on fields only)
• @Captor
• @Spy
• @InjectMocks
• @MockitoSettings

And the following Mockk features are supported for Kotlin codebases

• @MockKExtension
• @MockK
• @RelaxedMockK
• @SpyK
• @InjectMockks
• @MockKExtension.KeepMocks

Note that @MockKExtension.ConfirmVerification is not currently supported.

Since v1.0.3, the minimum supported release of Mockk is 1.9.1. Earlier accelerator releases require Mockk 1.12.1.

BeforeAlls and BeforeAfters

Static @BeforeAll and @AfterAll methods present a problem for mutation testing. Ideally, those test methods which execute the line of code a mutant is on should be run against it. But if a class has a @BeforeAll or @AfterAll annotation, the test methods cannot be run independently. The smallest executable “test unit” is the BeforeAll, followed by all the test methods in a class.

If the @BeforeAll is very expensive (e.g initializing a database), and the test methods are relatively cheap, then this may be the most efficient way to run the tests, even if only the very last test method run actually executes the mutant. But if the test methods are themselves expensive, or there is a large number of them, this can be very inefficient.

We recommend only using @BeforeAll and @AfterAll annotations if it is absolutely necessary to do so. In particular, test fixtures should not be made static simply because they are used within multiple test methods. This often has no measurable performance advantage, but does create the opportunity for difficult to debug test order dependencies.

When the accelerator encounters a @BeforeAll or @AfterAll, it takes a compromise approach. Test classes are split into “executable units” containing no more than three test methods each. The @BeforeAll and @AfterAlls will therefore be executed once every three tests instead of once per class. This improves performance in the majority of codebases we have examined.

It should be noted that tests with very expensive @BeforeAll methods are probably not suited to mutation testing, both for performance reasons and because they are likely that they interact with external resources (databases, file systems, network etc). Tests that depend on external resources are often not highly repeatable as they will fail when those resources are unavailable, and may be affected by state stored there.

MockkExtension support

The MockkExtension creates a similar problem to @BeforeAll and @AfterAll annotations. It’s default behaviour is to call unmockAll after each test class. When running through the accelerator, unmockAll is instead called after each test method. This ensures proper isolation when challenging mutants, and allows each test to be run independently.

Test classes using the MockkExtension that also include a @BeforeAll annotation will be not be accelerated. Doing so might result in test failures if the @BeforeAll initialises Mockks used by multiple methods in the class.

Installation

To you use the plugin you must first acquire a licence.

The licence file must be named arcmutate-licence.txt and placed at the root of the project.

The plugin itself should be placed on the classpath of the pitest build integration plugin.

E.g for maven

<plugin>
  <groupId>org.pitest</groupId>
  <artifactId>pitest-maven</artifactId>
  <version>1.16.0</version>
    <dependencies>
      <dependency>
        <groupId>com.arcmutate</groupId>
        <artifactId>pitest-accelerator-junit5</artifactId>
        <version>1.1.0</version>
      </dependency>
      <dependency>
        <groupId>org.pitest</groupId>
        <artifactId>pitest-junit5-plugin</artifactId>
        <version>1.0.0</version>
      </dependency>
    </dependencies>
</plugin>

Or for gradle

dependencies {
  pitest 'com.arcmutate:pitest-accelerator-junit5:1.1.0'
}