If a problem only occurs randomly once in every N times on average, how many tests do I have to perform to be certain that it's now fixed?

Question

I know that you can never be 100% sure, but is there a method to determine an appropriate number of tests?

Answer 1

I'm going to take a different approach than statistics (though I think the other response answers your actual question more directly). Any time I've encountered "a problem that only happens some of the time" as either QA or a support role it's been an investigative exercise about narrowing down why the event happens irregularly or in what situations it occurs.

Some (but certainly not all) points of investigation may be:

• Specific accounts or data.
• Differences in hosts/environments the applications or services are running on.
• Different versions of the application or service running on different hosts
• Certain days, dates, times of day or time zones.
• Certain users and their specific means of accessing the application (physical device, browser, network connection)

This sort of situation is where reproduction steps and other details from the person reporting the problem can be so valuable in resolving their issues. Telling a customer "your problem is fixed" when you're just making an educated guess can spiral in a negative direction if your experiment is based on incorrect assumptions. In my experience it's better to try to coach them about what information will help resolve their problem and how they can help you get it.

Answer 2

You must perform the same test equal number of times on both the unfixed version and the supposedly fixed version. You have to show that,

1. The test fails once in every N times randomly, on the unfixed version.
2. The same test passes every time, or at least fails less often, on the fixed version.

You have to show that the only difference between 1 and 2 is the "fix" itself, not any external or environmental factors.

If you only perform the test on the new, fixed version, it could very well be the case that the bug was caused by an unrelated environmental factor that simply doesn't exist in your test now.

Answer 3

I suppose this answer could help you

You need to decide first at what probability you want to "detect" the problem.

This is a nice example to why theoretical knowledge is necessary even for testers.

The simplified version:

• p is the probability for failure, 1/N in our case

• then the probability for success is 1-p

• and the probability to have N successful tries is (1-p)^N

• so the probability to have N successful tries and and then a failure would be 1-(1-p)^N

• extracting N and simplifying a bit assuming big enough N gives:

• −log(1−p)⋅N

(*) "log" is sometimes referred to (for example in calculators) as ln(x), loge(x) or log(x)

Answer 4

While I agree with the other answers saying "dig deeper", to answer the actual math question in the title:

If the issue occurs completely at random with probability p, then the chance if it occurring at least once in n trials is 1-(1-p)^n. Setting this to x (your confidence that the issue has been fixed) and solving for n gives you

n = log(1-x)/log(1-p)

---

So for example, if your issue occurs 1 out of 4 times, and you want to be 95% sure it's fixed (meaning you'll incorrectly identify it as fixed 1 out of 20 times!!), then

p = 0.25
x = 0.95
n = log(0.05)/log(0.75) ≈ 10.4

so you'd need to run 11 trials

---

The difference between my answer and @emrys57's is that mine assumes you know the probability, while theirs assumes you know some initial sequence of results. Presumably they should both give the same answer with a large enough initial sequence.

Answer 5

A problem has been observed that sometimes, in testing, produces errors. We don’t actually know the probability that it will produce an error on any given test run, because we can only do a finite number of tests on the broken system. If 1 represents a test pass, and 0 represents a test fail, we might have a sequence of results from multiple tests that looks like this:

011001010011

Representing 6 passes and 6 fails in 12 tests. This gives us a guess Pf of the probability of a failure of a test for the broken system. We assume that this probability isn’t changing with time. However, there will be a large uncertainty in the actual value of Pf since we cannot do very many tests.

Following these tests, we make a repair. We hope that this fixes the problem, but we’re not sure. We make more tests of the repaired system. If we have fixed the problem, the value of Pf measured after the repair should be 0. If we have not fixed the system, it should be the same as the value of Pf before the repair, unchanged Pf.

If we run some tests after the repair and one fails, we immediately know the repair failed. The question is, if no tests fail after repair, does that mean the repair worked?

Consider the case where we have results a zeroes (test fails) b ones (test passes) Including after repair: c ones (test passes)

The number of ways of arranging the a + b initial results is

Ntot = (a + b)! /(b! * a!)

In the case where Pf does not change yet all the last C results are ones, we need to arrange b - c ones in the first a + b - c results. The number of different ways of arranging these first a + b - c results is

Nsuc = (a + b - c)! / ( (b - c)! * a! )

These patterns are those from all the Ntot possible patterns where the last c results are all ones.

If the change didn’t actually repair the system, but really left its state the same, the results we observe are a random collection of zeroes and ones produced by chance, depending on the probability Pf that a single experiment will return a zero or one results. Given this, the chance that we will observe a pattern of a zeros and b ones with the last c results all ones is

Cran = Nsuc/Ntot

Or

Cran = (a + b - c)! * b! * a! / ((a +b)! * (b - c)!  * a!)

Or

Cran = (a + b - c)! * b! / ((a + b)! * (b - c)!)

Cran is the chance that we observe c test passes after we make a repair to the system, out of a total of a fails and b passes, but in fact we have changed nothing, and we see a sequence of passes after repair by happenstance.

As an example, with the pattern above before repair, and 6 ones (passes) after repair, Cran is slightly less than 5%. To reach a confidence of 99% that the repair has succeeded, you need 11 passes after repair.

There's a spreadsheet that can be copied to make this computation. I hope I have it all right, it is 15 years since I last worked this out. And, back then, it took me 15 years to first find the answer.

Answer 6

Let B mean "broken", F mean "fixed", E_k mean "error observed in k trials". You are saying that P(E_1|B)=1/N; the probability of seeing an error in a single observation, given that it's broken, is 1/N. Now, that itself is likely going to have some uncertainty, since likely your only way of measuring it will be by seeing how often it fails, and making an empirical estimate. However, if we take that as given, then applying Bayes' rule gives us that if our prior probability is P(B), then the posterior probability is P(~E_k|B)P(B)/P(~E_k).

For P(~E_k|B), we have P(~E_k|B)= P(~E_1|B)^k = (1-P(E_1|B))^k = (1-1/N)^k = ((N-1)/N)^k. For large N and k, we can approximate that as e^(-k/N).

For P(~E_k), we have P(~E_k) = P(~E_k|B)P(B)+P(~E_k|F)P(F). P(~E_k|F) =1 (If we've fixed it, we're guaranteed to see no errors). And P(F) is just 1-P(B).

Plugging that all in, we have

P(B|~E_k) ~= e^(-k/N)P(B)/(e^(-k/N)P(B)+1-P(B))

This can be made easier to read by setting X = e^(-k/N), Y = P(B). Then we have

XY/(XY+1-Y)

or

1-(Y-1)/(XY+1-Y)