I have a bug that is rarely reproduced, in about 1 case out of 5 attempts (20% reproducibility). Developers made a new version with the fix. How many times do I need to run a test to be sure that the bug is fixed? The bug is major or even critical, so it should be tested with extreme care. I cannot write an automated test because it's a complex scenario.
You can never be 100% sure that a rarely reproducible bug is fixed, you can only edge closer to 99.9999999...%
The simple way to monitor how confident you are in the fix is to run the reproduction steps over and over, logging how many times you (hopefully!) don't encounter the issue.
If the bug wasn't fixed, on the first test there is a 80% chance that the bug wasn't fixed but just didn't occur.
On the second test, there is now a 0.8 * 0.8 = 0.64 (64%) chance that the bug wasn't fixed but didn't occur.
On the third: 0.8^3 = 0.512 = 51.2% chance that the bug isn't fixed but just didn't occur.
You go on until you feel your confidence is good enough. You describe this as a potentially critical issue, so 99.5% may be your bottom confidence limit. For this, do 24 tests. This will give you a 0.8^24 = 0.004722 (0.47%) chance that the bug isn't fixed and just didn't occur.
For 99.9% confidence, you'd need to do 31 tests.
Probabilistically, Jake is certainly correct.
However, computers are deterministic machines, so when you say that you can only reproduce a problem 1/5 of the time, what you're really saying is that you're only actually creating the situation that leads to the problem 1/5 of the time. You need to talk to the developer, and get a better understanding of what the problem was, what caused it to occur, and what he did to fix it.
Because that's what I'd be much more concerned about. Was he able to track the problem down to a specific line of code/function/whatever, and fix that, and add test cases to make sure the problem doesn't happen again in the future, or did he do something to paper over the problem, maybe by adding another layer of recovery? Both of those will make the problem go away, but only one of them is fixing the problem. (And depending on the severity of the problem, and where the problem was found, and the nature of the problem, papering over the problem may be the right thing in the short term, but probably not in the long term.)
Ideally, the developer should be able to tell you what makes that 1/5 of the time different from the other 4/5 of the time, and you should be able to use that information to increase the chances of you hitting the problem to greater than 20%, possibly to 100%. And then you can be sure the problem is fixed, not almost sure the problem is fixed.
I like the other answers, but I would like to add automation to the mix.
I cannot write automated test because it's a complex scenario.
If you can do it manually (on a computer) you can probably automate it, but you do not have to create a maintainable automated test for this. I would expect most, if not all, scenarios are loopable with a record and playback tool.
In the past I have created UI tests that loop infinitely to reproduce issues that do not occur with clear defined steps. Issues that occur randomly.
I would try to make sure that the issue happens after at least X times or loops. That way developers know how long to run the automated test to verify that the issue was fixed.
This is certainly handy for issues that are related to memory leaks, database connections that are not always freed, or just random defects.
After the issue was fixed I would just loop the tests for several days for extra confirmation.
Still, it is wise to do a root-cause analysis, think about whether other locations might have a similar problem, and also tackle those. Nothing is worse than fixing it in one location and then having your client start complaining about another one.
This is a great example of an almost criminal use of statistics.
Reminds me of Cargo Cult coding ( https://en.wikipedia.org/wiki/Cargo_cult_programming ).
Unless your trials are random (meaning mindless) and they weren't, your use of statistical reasoning is almost certainly wrong.
The math has to follow the model, and you don't seem to have a good understanding of the model of the problem. (If you did, you wouldn't be asking your question). The obvious answer is beta testing. Assuming you want to know how much testing is required prior to beta testing, then the answer is you need to do "sufficient" testing so that you're confident that the frequency of occurrence is much reduced (essentially gone).
Given that random noise typically scales with √n, and you'd have to do about 10 (random) trials to be fairly sure the bug rate wasn't 20% (or more) then do 100. Frankly, if you don't try it with a number (how many? IDK, maybe 10 or 20) machines/network configs (with which you are able to demonstrate the bug occurs (in the old code)) and at least 20 or 50 replicates on each, I'd not be very comfortable with the claim that it's been fixed. Again, without a good model of the problem, any confidence you have that its been fixed is mostly wishful thinking. Sorry. We lack the ability to reliably determine the reliability in all but the most simple code.
The other answers I've seen assume a pure cut and paste scenario where "bad code" is cut out and replaced with "good code". That almost never happens. Patches usually address multiple (and often interrelated) problems. Bugs that are confined to one line of source code generally do not cause "rare" occasional bugs. One final comment. I'd never employ a programmer who believes a 20% frequency rate is a rare problem. Lets say 20% of your friends die every year. Rare? Not! Sure it's a moot point, but my idea of rare is 1 in 1000 or less, not 20%.
Experience taught me the environment is essential in such cases. You need to understand the fix, what problem did the developer address. For instance, maybe the developer simply made the code less restrictive. Or more defensive - e.g. return earlier in the flow in case of unexpected input. If this is the case, ask him to give you a version without the fix but with logs in the area where he put the shield. Check the logs carefully before and after the fix. Then you will learn how much testing is enough.
I don't know if you're speaking of a purely software system, or one that processes input from electronics and sensors, but either way, the answer is unit testing and white box testing. Your failure mode either occurs inside the expected inputs or outside.
I'm going to assume you're already unit testing using a range of expected inputs as well as inputs outside the boundary conditions in which your code is designed to run. Inputs beyond boundary conditions should be handled by validation. That may be the fix right there, or it may point to some upstream device or data not outputting data within its design spec. Either way, you want to validate input on your code.
If failure modes still occur while sweeping through the valid input ranges, then you can find them using the original code.
Once you reproduce the combination of conditions under which a failure occurs (and there may be more than one), you can then determine if those conditions present inputs where the original code would have failed and your fix is effective via whitebox examination of the code. If your fix addresses all failure modes found in your unit testing, you've demonstrated it is sound and effective. If it doesn't address all failure modes, there is more work to do.
I'm working on Windows based application and as per my experience I came across this situation many times. For few bugs some time simulation was medium and sometime low.
So the best way to verify whether the bug got resolved is try to understand the bug i.e Discuss it with developer try to understand the cause of that bug (after giving the bug demo) and once it is resolved ask him (Developer) about the solution and then perform the reproduction(simulation) steps for few times and also some scenario related to latest fix.
Lets consider following mentioned example, While testing one of the video surveillance application I came across one crash. It was simulating when user was trying to listen live audio for specific webcam. But simulation was very rare. Some how I gave the problem demo to one of the developer with proper stack trace.We tried to get the exact simulation through stack trace but we didn't as simulation was very rare. So at last through stack trace developer understood the problem and cause was related to start state of audio device. So he applied some lock at start and stop audio device. After understanding the solution I was left with scenarios to test whether application is getting hung while adding, removing, starting and stopping webcam.