I have a bug that is rarely reproduced, in about 1 case out of 5 attempts (20% reproducibility). Developers made a new version with the fix. How many times do I need to run a test to be sure that the bug is fixed? The bug is major or even critical, so it should be tested with extreme care. I cannot write an automated test because it's a complex scenario.
1Sounds like Russel's Teapot made it to software engineering as well ;)– GallifreyanMar 31, 2017 at 19:42
2I had a project a few years back where the ethernet chip on a device with custom firmware on a custom PCB would lock up, randomly, on average once every 3 months of continuous operation, except the project involved 100 devices so overall the average was daily. It was also site-specific, we never got a device to hang in the shop, and we tested 20 devices continuously for an entire year. The only way to debug was to attach a hardware debugger when the problem happened. A power cycle would fix it. Access required a week of planning and a licensed, union forklift operator. ...– Jason CApr 1, 2017 at 0:09
2... And every time we didn't get any info in our one shot with a debugger, it cost a few thousand $ in labor, access permits and fees, and we had to go through all the red tape of planning a whole new access visit to try again. To this day, we never solved the problem. The best working theory was site-specific EMI combined with an unfortunate PCB trace layout. In the end I just made the device reboot itself if it didn't see network data for a while. A frustrating ending to a nightmarish project that still sometimes keeps me awake at night.– Jason CApr 1, 2017 at 0:10
4@JasonC The amount of duct tape and chicken wire that hold together billion dollar systems is mind boggling.– corsiKa ♦Apr 2, 2017 at 4:39
Closely related: sqa.stackexchange.com/q/28135/17060– WildcardJul 1, 2017 at 4:22
You can never be 100% sure that a rarely reproducible bug is fixed, you can only edge closer to 99.9999999...%
The simple way to monitor how confident you are in the fix is to run the reproduction steps over and over, logging how many times you (hopefully!) don't encounter the issue.
If the bug wasn't fixed, on the first test there is a 80% chance that the bug wasn't fixed but just didn't occur.
On the second test, there is now a 0.8 * 0.8 = 0.64 (64%) chance that the bug wasn't fixed but didn't occur.
On the third: 0.8^3 = 0.512 = 51.2% chance that the bug isn't fixed but just didn't occur.
You go on until you feel your confidence is good enough. You describe this as a potentially critical issue, so 99.5% may be your bottom confidence limit. For this, do 24 tests. This will give you a 0.8^24 = 0.004722 (0.47%) chance that the bug isn't fixed and just didn't occur.
For 99.9% confidence, you'd need to do 31 tests.
1Thank you for that edit! Amazing how far a simple typo can mislead... Mar 30, 2017 at 15:24
I guess 16 repeats would be enough. 97% is enough probability. Thank you very much. Mar 30, 2017 at 16:04
You're welcome. For more information on intermittent bugs, this blog post has some wonderful points Mar 30, 2017 at 16:10
16I'd highlight this all assumes the 1/5 is a strong data point, that you know the bug really does happen 20% of the time, which seems a little iffy. As a tester, I'd be really pushing for what was identified as the root cause of the bug, and knowing that, can I reproduce it 100% of the time? Does the code change make it so I can no longer reproduce?– ernieMar 30, 2017 at 21:13
3The problems with this approach are two-fold: 1) If you don't know how to trigger the bug, then the method of testing might completely miss the trigger, hence modeling it as an independent runs of an experiment with a 20% probability is problematic, and 2) it is unlikely that the attempted bug-fix had no effect at all, the main danger is that they would have fixed some of the scenarios that lead to the bug but neglected some edge cases that still result in it. In that case, the frequency of the bug would have decreased, and merely showing that it doesn't occur 20% of the time shows little. Apr 1, 2017 at 16:29
Probabilistically, Jake is certainly correct.
However, computers are deterministic machines, so when you say that you can only reproduce a problem 1/5 of the time, what you're really saying is that you're only actually creating the situation that leads to the problem 1/5 of the time. You need to talk to the developer, and get a better understanding of what the problem was, what caused it to occur, and what he did to fix it.
Because that's what I'd be much more concerned about. Was he able to track the problem down to a specific line of code/function/whatever, and fix that, and add test cases to make sure the problem doesn't happen again in the future, or did he do something to paper over the problem, maybe by adding another layer of recovery? Both of those will make the problem go away, but only one of them is fixing the problem. (And depending on the severity of the problem, and where the problem was found, and the nature of the problem, papering over the problem may be the right thing in the short term, but probably not in the long term.)
Ideally, the developer should be able to tell you what makes that 1/5 of the time different from the other 4/5 of the time, and you should be able to use that information to increase the chances of you hitting the problem to greater than 20%, possibly to 100%. And then you can be sure the problem is fixed, not almost sure the problem is fixed.
11Exactly right. The developer who fixes it should also be able to improve your "steps to reproduce" so that you can produce the issue 100% of the time on the old version, and 0% of the time on the new version.– WildcardMar 30, 2017 at 18:37
23A lot of the time this is right, but not always. Synchronisation bugs are the big exception. When the bug is caused by a race condition, it may be iimpossible to very difficult to rig the race to trigger the bug all the time.– LelielMar 30, 2017 at 20:42
3+1. The developer now knows more about why the bug was actually occurring. They should be able to guide you. If nothing else, they should be able to confirm that the error was indeed random (as opposed to being caused by a particular user behavior), which would then make Jake's answer valid. Mar 30, 2017 at 20:59
1Code can be nondeterministic, AND you can have race conditions, AND real world inputs can be nondeterministic. Mar 31, 2017 at 14:28
1If they can't explain how the problem happens, then they need to improve their tracing/debug capabilities. Mar 31, 2017 at 18:27
I like the other answers, but I would like to add automation to the mix.
I cannot write automated test because it's a complex scenario.
If you can do it manually (on a computer) you can probably automate it, but you do not have to create a maintainable automated test for this. I would expect most, if not all, scenarios are loopable with a record and playback tool.
In the past I have created UI tests that loop infinitely to reproduce issues that do not occur with clear defined steps. Issues that occur randomly.
I would try to make sure that the issue happens after at least X times or loops. That way developers know how long to run the automated test to verify that the issue was fixed.
This is certainly handy for issues that are related to memory leaks, database connections that are not always freed, or just random defects.
After the issue was fixed I would just loop the tests for several days for extra confirmation.
Still, it is wise to do a root-cause analysis, think about whether other locations might have a similar problem, and also tackle those. Nothing is worse than fixing it in one location and then having your client start complaining about another one.
Where does it say the test scenario does not involve real-world/off-line steps? Think of an ecosystem where physical labels are printed and scanned as an example that is difficult to automate :) Think of a testing scenario that involves a production system in which test data, or results of a failed test, have to be manually cleaned up. Think of software that initiates transactions (in the worst case literal bank transactions) in some other system as part of its tests... Mar 31, 2017 at 14:33
2@rackandboneman Changed my texts a bit. Yes, I falsely assume it is a process on a computer. Still I would like to keep my answer as to trigger testers to think about automation and make an issue repeatable per automation. Even some true manual processes can be automated. I have seen model trains be used to test wireless credit-cards transactions :) just try to think out of the box. Mar 31, 2017 at 14:43
This is a great example of an almost criminal use of statistics.
Reminds me of Cargo Cult coding ( https://en.wikipedia.org/wiki/Cargo_cult_programming ).
Unless your trials are random (meaning mindless) and they weren't, your use of statistical reasoning is almost certainly wrong.
The math has to follow the model, and you don't seem to have a good understanding of the model of the problem. (If you did, you wouldn't be asking your question). The obvious answer is beta testing. Assuming you want to know how much testing is required prior to beta testing, then the answer is you need to do "sufficient" testing so that you're confident that the frequency of occurrence is much reduced (essentially gone).
Given that random noise typically scales with √n, and you'd have to do about 10 (random) trials to be fairly sure the bug rate wasn't 20% (or more) then do 100. Frankly, if you don't try it with a number (how many? IDK, maybe 10 or 20) machines/network configs (with which you are able to demonstrate the bug occurs (in the old code)) and at least 20 or 50 replicates on each, I'd not be very comfortable with the claim that it's been fixed. Again, without a good model of the problem, any confidence you have that its been fixed is mostly wishful thinking. Sorry. We lack the ability to reliably determine the reliability in all but the most simple code.
The other answers I've seen assume a pure cut and paste scenario where "bad code" is cut out and replaced with "good code". That almost never happens. Patches usually address multiple (and often interrelated) problems. Bugs that are confined to one line of source code generally do not cause "rare" occasional bugs. One final comment. I'd never employ a programmer who believes a 20% frequency rate is a rare problem. Lets say 20% of your friends die every year. Rare? Not! Sure it's a moot point, but my idea of rare is 1 in 1000 or less, not 20%.
Experience taught me the environment is essential in such cases. You need to understand the fix, what problem did the developer address. For instance, maybe the developer simply made the code less restrictive. Or more defensive - e.g. return earlier in the flow in case of unexpected input. If this is the case, ask him to give you a version without the fix but with logs in the area where he put the shield. Check the logs carefully before and after the fix. Then you will learn how much testing is enough.
I don't know if you're speaking of a purely software system, or one that processes input from electronics and sensors, but either way, the answer is unit testing and white box testing. Your failure mode either occurs inside the expected inputs or outside.
I'm going to assume you're already unit testing using a range of expected inputs as well as inputs outside the boundary conditions in which your code is designed to run. Inputs beyond boundary conditions should be handled by validation. That may be the fix right there, or it may point to some upstream device or data not outputting data within its design spec. Either way, you want to validate input on your code.
If failure modes still occur while sweeping through the valid input ranges, then you can find them using the original code.
Once you reproduce the combination of conditions under which a failure occurs (and there may be more than one), you can then determine if those conditions present inputs where the original code would have failed and your fix is effective via whitebox examination of the code. If your fix addresses all failure modes found in your unit testing, you've demonstrated it is sound and effective. If it doesn't address all failure modes, there is more work to do.
I'm working on Windows based application and as per my experience I came across this situation many times. For few bugs some time simulation was medium and sometime low.
So the best way to verify whether the bug got resolved is try to understand the bug i.e Discuss it with developer try to understand the cause of that bug (after giving the bug demo) and once it is resolved ask him (Developer) about the solution and then perform the reproduction(simulation) steps for few times and also some scenario related to latest fix.
Lets consider following mentioned example, While testing one of the video surveillance application I came across one crash. It was simulating when user was trying to listen live audio for specific webcam. But simulation was very rare. Some how I gave the problem demo to one of the developer with proper stack trace.We tried to get the exact simulation through stack trace but we didn't as simulation was very rare. So at last through stack trace developer understood the problem and cause was related to start state of audio device. So he applied some lock at start and stop audio device. After understanding the solution I was left with scenarios to test whether application is getting hung while adding, removing, starting and stopping webcam.