Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

This is part of my quest how to test my tests and test data. I looked at the techniques that would help improve my test cases. While code/branch coverage can tell me whether my test suites cover my code base sufficiently, it will not tell me whether my test oracles are able to detect all faults. So, here comes mutation testing with help:

Mutation testing involves modifying a program's source code or byte code in small ways. A test suite that does not detect and reject the mutated code is considered defective.

In extreme case it will detect useless assertions 1==1 in your tests :-)

There are tools for performing mutation testing (in Java and other programming languages), some well integrated with xUnit frameworks. However, configuring those tools and interpreting their results seems time-consuming to me, and the tools might return many false negatives (called equivalent mutants) that I must filter out manually. Hence, I'm curious whether it is worth investing more time on learning that technique.

  1. Have you found those tools, or mutation testing in general, useful for improving your tests?
  2. What kind of deficiencies have you found in your tests with mutation testing?

I've read about some hands-on experiences with this technique but no details what deficiencies were found in real test cases and how serious they were.

share|improve this question
    
I found the following an example of mutation testing use. We have method foo() that calls method bar() on object X, if a certain condition has been satisfied. In the test, we check with interaction on mock of object X happened when condition has been satisfied, but we do not check whether there was no such interaction when condition was not satisfied. A mutant removes (or reverse condition in) "if" statement, while tests are still green. This suggests two options to follow: (a) either "if" statement is unnecessary or (b) a new test checking lack of interaction can be added. –  dzieciou Nov 27 '12 at 12:32

3 Answers 3

up vote 5 down vote accepted

I'm the author of one of the tools for Java (http://pitest.org).

I have found mutation testing useful in practice, working on large legacy corporate code bases and smaller test driven projects.

I've seen a lot of dismissals of the value of mutation testing such as Bj Rollison's answer. I generally don't disagree with the points he makes, however some implicit assumptions are being made about how mutation testing can/should be used.

Mutation testing is useful (in my experience) when performed in the following ways

1. From the outset of a small/medium project.

In this scenario mutation testing supports you as you code. You can realistically aim to achieve a high mutation score.

Mutation testing will highlight gaps in your test suite (examples I keep seeing are interaction focused tests that don't check that methods actually return the value they're meant to calculate, unchecked boundaries, and lots of instances where the TDD process you claim to be following clearly wasn't followed as you can take methods calls out without a test failing).

A surviving mutation will result in you doing one of three things

a) Writing a new test

b) Deleting some code

c) Rewriting some code

Often the outcome is b) or c).

b) is self explanatory - you have some useless code that can be removed, c) is less obvious and relates to equivalent mutations. I've found that often code that can be mutated to an equilvaent mutation can be better expressed by code that can't be mutated to an equivalent mutation. The result is usually easier to understand and maintain.

Some equivalment mutations will always remain however - usually these are from performance concerns within the code.

The definition of what constitutes a small/medium project is a little tricky to tie down - but basically small enough that the analysis completes in a time low enough to be acceptable as part of you coding feedback loop. Fortunately mutation testing tools have improved their performance by orders of in recent years. (see http://pitest.org/java_mutation_testing_systems/ for some discussion of why/how).

2. To support new development within an existing large code base

The same as above but the analysis is limited to new code. It can be useful both while developing, and part of pair review etc. It finds test suite gaps, but can also generate some useful discussion about the code.

3. To support changes to existing code within an existing larger code base

In this scenario a slice of a codebase of interest is picked out for analysis - often to support a refactoring.

As before you might end up adding tests, but probably wouldn't bother deleting or rewriting code as it is likely to be replaced anyway. Once all the surviving mutations have been investigated you have however gained an accurate understanding of the risk involved in refactoring.

They way most people seem to assume that mutation testing should be used is ..

4. Try and achieve a 100%/85%/75% etc mutation score for an existing project

I've never tried this, and wouldn't expect it to deliver much value.

If the code base is large analysis times will be long, and the number of surviving mutations likely to be high. Trying to assess each one would be prohibitively time consuming (javalanche is meant to provide some help in this space - I've not tried it myself though, and gather it comes with a high performance cost).

Mutation testing could perhaps still be used to assess the overall quality of and existing suite. A random sample of surviving mutation could be taken and each assesed to see if it equivalent - this could then be used to determine the likely number of surviving mutations that repesent test suite deficiencies. This is not something I have tried myself however.

share|improve this answer
    
Have you tried Pitest on real project? How much time it took you to investigate survived mutations? How serious were defects you found? –  dzieciou Nov 26 '12 at 15:14
    
Yes, several real projects as described above. The main point of mutation testing isn't to find defects - it's to assess the quality of a test suite. I've found lots of test suite deficiencies with mutation testing and lots of unnecessary code. I have however found some defects via mutation testing - mainly incorrect conditional boundaries. –  henry Nov 26 '12 at 15:20
3  
In terms of how much time it took to assess the surviving mutations . . . difficult to say. This was generally part of the normal development process and not measured separately. This is the point I was trying to drive home above, applying it as an after the fact quality assessment phase is going to be difficult. Where I've had success it has been part of the development process. –  henry Nov 26 '12 at 15:29
    
Thanks @henry. Do you have any case study with examples of deficiences you found in your tests? Something that shows: here is the mutation that survived, here's my test code, and here's how I corrected my test code. –  dzieciou Nov 26 '12 at 17:03
3  
Henry, you make some very good arguments in favor of mutation testing for testing the effacacy of unit tests on private builds. I was speaking from a higher level perspective of integration testing of components and system level testing, and should have clarified that. –  Bj Rollison Nov 27 '12 at 2:15

Mutation testing has been used in some small systems e.g.: see article "An intuitive approach to determine test adequacy in safety-critical software". Here's the abstract:

Safety-critical software must adhere to stringent quality standards and is expected to be thoroughly tested. However, exhaustive testing of software is usually impractical. The two main challenges faced by a software testing team are generation of effective test cases and demonstration of testing adequacy.

This paper proposes an intuitive and conservative approach to determine the test adequacy in safety-critical software. The approach is demonstrated through a case study: the core temperature monitoring system of a nuclear reactor. We combine conservative test coverage of unique execution path test cases, and the results from mutation testing to determine the test adequacy.

Although mutation testing is a powerful technique, the difficulty in identifying equivalent mutants has limited its practical utility. To gain confidence on the computed test adequacy:

  1. faults during mutation testing must be induced at all possible execution paths of the code,
  2. properties of unkilled mutants must be studied, and
  3. all equivalent mutants must be detected.

In this regard;

  • results of static, dynamic and coverage analysis of the mutants is presented, and
  • a technique to identify the likely equivalent mutants is proposed.
share|improve this answer
1  
Welcome to SQAT.SE - could you give an executive summary? Links to articles (especially "paid for") are not very helpful in the SE context... –  Andrew Nov 30 '12 at 6:28

Mutation testing is generally interesting from an academic perspective. I have never seen or heard of anyone using mutation testing on software that is delivered to customers on a schedule. The time and expense involved in mutation testing is huge compared to the perceived value returned. In complex projects with several developers contributing to the product mutation testing of even a small component can block continuous integration cycles. In general, mutation testing is probably one of the most expensive solutions to a goal of improving test case effectiveness (assuming that you have a problem of ineffective test cases).

Mutation testing may be useful in training, or smaller projects with limited participants and no timelines, but it is generally more complicated than flipping a single bit and analyzing the results of a subset of tests.

Also, it is a faulty assumption that an oracle (human or automated) will detect all faults. Automated oracles can only detect problems they are programmed to check for.

share|improve this answer
    
So, it looks like using traditional test design techniques to cover important partitions of values/states would serve better for improving test quality. –  dzieciou Nov 25 '12 at 22:24
2  
It is unclear that the kinds of bugs created by mutation are representative of what developers will create. –  user246 Nov 26 '12 at 1:55
    
@User246 Isn't this the same story as setting up static analysis tool like FindBugs? Many programmers make mistakes causing NullPointException, so you set up FindBugs to be sensitive about those. Similarly, for mutation testing you may turn-on mutation types, which are representative (if you obviously now which of them are...) –  dzieciou Nov 26 '12 at 6:49

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.