What are the best practices for error seeding

Question

Where I work we use an offshore company to augment several internal departments. I work in the QA team, and concerns have been aired about the level of confidence we can have in the test work we have been assigning to workers at this company.

I have suggested that it may be a good idea to use error seeding to get a metric of our confidence level in code that has been tested by this company, and compare it with the same metric as used internally.

This is a technique I learned about on an ISEB testing course, however I have not met many people who have used it in a real-world environment, and so would like some input as to what best practices may be. The following are things which I have thought of, but obviously it would be better to learn from the experience of someone who has tried this before.

• Have developers seed multiple (n) bugs per testable item
• Have developers track these bugs in a system that QA is not able to access
• Alert the tester for said testable item to how large n is (approx? absolute?)
• For each bug that tester raises, alert them as to whether the bug was seeded or not

My reasoning is that if we make this metric visible and transparent, it will also give testers feedback as to how rigorous their testing is.

I would welcome input as to any practical issues that may make this difficult or even unworkable. If this is implemented I shall leave an answer updating everyone on my own experience with this.

Answer 1

I agree with Joe's assertion that metrics can be badly misused and counterproductive.

That said, error seeding can be a useful way to answer the right question.

After we develop a test plan, we generally assume two things:

1. The test plan is unlikely to uncover every error in the system. (If the system is large and complex, we can assume some undiscovered errors will remain.)

2. Still, the test plan should be able to uncover many errors in the system. (Otherwise, it's not a very well-designed test plan.)

Let E = the total number of error in system, D = the number of errors discovered by the test plan, and U = the total number of undiscovered errors. We know that E = D + U, but we can never be sure what E or U are. However, we'd like to think E/D is close to 1.

So, how do we calculate E/D - our error detection rate? Because we'll never know E or U, we can't! But we can inject faults into the code, and then run the test plan. By seeing how many of the known errors get detected, we can calculate an error detection rate in a controlled environment. Knowing if this controlled error detection rate is high or low might prove instructive.

Some notes and pitfalls:

• This method is designed to collect meaningful data where prewritten test plans are run. They provide an indication about whether or not these plans are likely to catch errors, or simply lulling testers into a false sense of security.
• This will only work if the seeded errors are plausible, and striking a balance between being well-hidden and detectable. If the errors are too easy to find (or too hard to find), the resulting data will be neither accurate nor useful. Therefore, it takes a fair amount of expertise to seed the errors. (One possibility would be to remove a divide-by-zero check. If no test case in the test plan finds the error, then perhaps that points to an area where the test plan could be improved.)
• The people injecting the faults should be unfamiliar with the test plan. Moreover, if the test plan isn't already written, those who write the test plan should be completely unaware of the injected bugs. Otherwise, the two groups will be engaged in a "chess match" of sorts, and the data won't provide an accurate indication of the test plan's effectiveness.
• This isn't meant to be an ongoing exercise, so you needn't worry about testers "stopping" after the seeded bugs are found. This technique isn't used to test the system, it's used to test the test plan.

Joe is right to point out that this technique is not worth doing, unless it is done correctly. That said, it still has its place in the software testing realm, particularly in organizations that rely heavily on pre-written or scripted tests.

Answer 2

I haven't been part of a project where fault seeding was used.

I have, however been part of projects where new metrics were introduced - which is essentially what you are proposing here. If my experience is any predictor, you should expect that people will get better at specifically whatever it is you measure, at the expense of everything else.

Here, you might expect the testers would get better at finding the seeded errors, and worse at other things. Those other things might include - helping others, documentation, finding non-seeded bugs, collaborating with developers, etc, etc.

You might also find that, once the seeded bugs are found, testers stop testing.

And you might find morale issues among the testers (and perhaps developers and others as well).

With any metrics program, consider carefully the unintended consequences and side-effects, on the testers, developers, and others.

If this is implemented I shall leave an answer updating everyone on my own experience with this.

I'd love to see how this worked out.

Answer 3

Disclaimer: I do not have any direct experience with this.

Researchers (e.g. Atif Memon at the University of Maryland) use fault seeding to measure the effectiveness of testing techniques. In that context, the practice makes sense. I assume you propose to use seeding as an experiment, not as an on-going process.

You mentioned rigorous testing. In a rigorous experiment, you decide ahead of time how you plan to interpret your results. Everyone involved in this experiment -- you, your QA team, your developers, and the outsourced team -- will have their own biases that will impact how they respond to the experiment's measurements. You can reduce the effect of those biases by agreeing on how you will collect and analyze the measurements before you begin the experiment. This includes agreeing on a sample size that is large enough to be statistically significant.

Edit

In addition to measuring the number of found bugs and missed bugs, it may also be interesting to measure the number of false positives, i.e. the number of reports that turns out not to be bugs.