Mutation testing: is it useful for assessing test quality?

Question

This is part of my quest how to test my tests and test data. I looked at the techniques that would help improve my test cases. While code/branch coverage can tell me whether my test suites cover my code base sufficiently, it will not tell me whether my test oracles are able to detect all faults. So, here comes mutation testing with help:

Mutation testing involves modifying a program's source code or byte code in small ways. A test suite that does not detect and reject the mutated code is considered defective.

In extreme case it will detect useless assertions 1==1 in your tests :-)

There are tools for performing mutation testing (in Java and other programming languages), some well integrated with xUnit frameworks. However, configuring those tools and interpreting their results seems time-consuming to me, and the tools might return many false negatives (called equivalent mutants) that I must filter out manually. Hence, I'm curious whether it is worth investing more time on learning that technique.

1. Have you found those tools, or mutation testing in general, useful for improving your tests?
2. What kind of deficiencies have you found in your tests with mutation testing?

I've read about some hands-on experiences with this technique but no details what deficiencies were found in real test cases and how serious they were.

Answer 1

I'm the author of one of the tools for Java (http://pitest.org).

I have found mutation testing useful in practice, working on large legacy corporate code bases and smaller test driven projects.

I've seen a lot of dismissals of the value of mutation testing such as Bj Rollison's answer. I generally don't disagree with the points he makes, however some implicit assumptions are being made about how mutation testing can/should be used.

Mutation testing is useful (in my experience) when performed in the following ways

1. From the outset of a small/medium project.

In this scenario mutation testing supports you as you code. You can realistically aim to achieve a high mutation score.

Mutation testing will highlight gaps in your test suite (examples I keep seeing are interaction focused tests that don't check that methods actually return the value they're meant to calculate, unchecked boundaries, and lots of instances where the TDD process you claim to be following clearly wasn't followed as you can take methods calls out without a test failing).

A surviving mutation will result in you doing one of three things

a) Writing a new test

b) Deleting some code

c) Rewriting some code

Often the outcome is b) or c).

b) is self explanatory - you have some useless code that can be removed, c) is less obvious and relates to equivalent mutations. I've found that often code that can be mutated to an equilvaent mutation can be better expressed by code that can't be mutated to an equivalent mutation. The result is usually easier to understand and maintain.

Some equivalment mutations will always remain however - usually these are from performance concerns within the code.

The definition of what constitutes a small/medium project is a little tricky to tie down - but basically small enough that the analysis completes in a time low enough to be acceptable as part of you coding feedback loop. Fortunately mutation testing tools have improved their performance by orders of in recent years. (see http://pitest.org/java_mutation_testing_systems/ for some discussion of why/how).

2. To support new development within an existing large code base

The same as above but the analysis is limited to new code. It can be useful both while developing, and part of pair review etc. It finds test suite gaps, but can also generate some useful discussion about the code.

3. To support changes to existing code within an existing larger code base

In this scenario a slice of a codebase of interest is picked out for analysis - often to support a refactoring.

As before you might end up adding tests, but probably wouldn't bother deleting or rewriting code as it is likely to be replaced anyway. Once all the surviving mutations have been investigated you have however gained an accurate understanding of the risk involved in refactoring.

They way most people seem to assume that mutation testing should be used is ..

4. Try and achieve a 100%/85%/75% etc mutation score for an existing project

I've never tried this, and wouldn't expect it to deliver much value.

If the code base is large analysis times will be long, and the number of surviving mutations likely to be high. Trying to assess each one would be prohibitively time consuming (javalanche is meant to provide some help in this space - I've not tried it myself though, and gather it comes with a high performance cost).

Mutation testing could perhaps still be used to assess the overall quality of and existing suite. A random sample of surviving mutation could be taken and each assesed to see if it equivalent - this could then be used to determine the likely number of surviving mutations that repesent test suite deficiencies. This is not something I have tried myself however.

Answer 2

Mutation testing is generally interesting from an academic perspective. I have never seen or heard of anyone using mutation testing on software that is delivered to customers on a schedule. The time and expense involved in mutation testing is huge compared to the perceived value returned. In complex projects with several developers contributing to the product mutation testing of even a small component can block continuous integration cycles. In general, mutation testing is probably one of the most expensive solutions to a goal of improving test case effectiveness (assuming that you have a problem of ineffective test cases).

Mutation testing may be useful in training, or smaller projects with limited participants and no timelines, but it is generally more complicated than flipping a single bit and analyzing the results of a subset of tests.

Also, it is a faulty assumption that an oracle (human or automated) will detect all faults. Automated oracles can only detect problems they are programmed to check for.

Answer 3

Mutation testing has been used in some small systems e.g.: see article "An intuitive approach to determine test adequacy in safety-critical software". Here's the abstract:

Safety-critical software must adhere to stringent quality standards and is expected to be thoroughly tested. However, exhaustive testing of software is usually impractical. The two main challenges faced by a software testing team are generation of effective test cases and demonstration of testing adequacy.

This paper proposes an intuitive and conservative approach to determine the test adequacy in safety-critical software. The approach is demonstrated through a case study: the core temperature monitoring system of a nuclear reactor. We combine conservative test coverage of unique execution path test cases, and the results from mutation testing to determine the test adequacy.

Although mutation testing is a powerful technique, the difficulty in identifying equivalent mutants has limited its practical utility. To gain confidence on the computed test adequacy:

1. faults during mutation testing must be induced at all possible execution paths of the code,
2. properties of unkilled mutants must be studied, and
3. all equivalent mutants must be detected.

In this regard;

• results of static, dynamic and coverage analysis of the mutants is presented, and
• a technique to identify the likely equivalent mutants is proposed.