Extreme Mutation Testing

Requires pitest 1.15.0 or above.

Provided by the arcmutate-base plugin.

Background

Extreme mutation testing introduces mutants which remove entire method bodies.

For example

public List<Integer> foo(int a, int b) {
  if (a >= 42) {
     return asList(a,b);
  }
  return asList(a);
}

Would be mutated to

public List<Integer> foo(int a, int b) {
  return Collections.emptyList();
}

This approach generates far fewer mutants than traditional mutation testing, thereby reducing analysis times.

Although the mutations produced by extreme mutation testing are very coarse grained, research indicates they are still effective at detecting psuedo-tested methods (methods that are covered by tests, but do not have their behaviour verified).

Extreme mutation testing provides little feedback on the analysed code and tests, but we believe it can still fill a useful gap. It provides an easy way to introduce mutation testing to an existing project, allowing an assessment of the entire codebase to be made without overloading the user with information or requiring long analysis times. As the code and tests are improved, other operators can be gradually added, or a traditional analysis can be performed on only modified code using arcmutate’s pull request integration.

Subsumption

The extreme mutation operators are very unstable, so can be easily detected by most tests that perform some form of validation step such as an assertion.

If other mutation operators have been configured, it is therefore to be expected that arcmutate will generate few, is any, extreme mutants.

For example, for the code shown earlier, the default pitest operators would generate the following mutants

public List<Integer> foo(int a, int b) {
  if (a >= 42) {
    return Collections.emptyList(); // mutated by empty returns mutator
  }
  return asList(a);
}

public List<Integer> foo(int a) {
  if (a >= 42) {
     return asList(a,b);
  }
  return Collections.emptyList(); // mutated by empty returns mutator
}

public List<Integer> foo(int a) {
  if (a > 42) { // mutated by conditional boundaries mutator
     return asList(a,b);
  }
  return asList(a);
}

The extreme mutant would be killed by a test that provided any input and asserted that foo returned a non empty list.

The same test would only kill the first mutant if a was >= 42.

The second mutant requires a to be < 42.

The third mutant requires a to be exactly equal to 42.

Each of these mutants therefore subsumes the extreme mutant as they are killed by a smaller set of possible tests.