What should be reported when a bug only happens 50% of the time?

Question

So, let's say I'm running some tests and run across some odd behavior that is a deviation from expected results, however, I quickly run the test again and the bug is not reproduced. I begin to question what's going on and run, say, 20 iterations of the test. I find that the unexpected behavior only occurs 9/20 times while the other 11/20 times the application runs smoothly.

I've encountered this while on-the-job in QA and when I ask about it I find that my colleagues, when shown the test, do not seem very worried about this kind of behavior. They seem happy to simply pass the test!

This has been IRKING me. What should I report? My instinct was to fail the test and go into a deeper dive around the issue. However, since this could be an issue of priority, maybe the Product Owner just wants to get other things done first. But what's wrong with logging a trivial bug? What kind of reaction should I have to this kind of a situation?

Answer 1

A bug that is only a bug 50% of the time is still a bug.

I would still report/document it as a regular bug, but with an "intermittent" label - letting to know both developers and management about the bug, and that it is not reproduced all the time.

This way the issue would get its priority by the management, and the mentioned people may generate some ideas about making the bug reproducible 100% of the time. You would also let this "IRKING" partially go away having the fact that you observed the issue recorded.

The important thing is to document as much information about the facts and observations when the bug is reproduced and when is not - record as much as possible - screenshots, logs, stacktrace, current theories and ideas. Think about things that vary from one test execution to another - test order, different pre- and post- conditions, network delays, user access control, etc.

Answer 2

Try to get a reliably-failing test case, but at the end of the day, it's still a bug. Make sure to state that e.g. "hey, this only fails ~1/2 the time".

(Note that one way to dance around the issue if it's impolitic is to write a new test case that runs the test e.g. 100 times.)

I once reported an HW bug that occurred ~1-in-100k times on ~1/10 devices... (And I found it on iteration 9986/10k, and even that was only because I forgot to stop it before I left for the weekend...) The root cause was one of those "if the stars line up" bugs - a write in a 1 clock cycle window would tristate an output for a bit, and everything would be OK as long as said output retained its value for long enough.

Answer 3

A bug that cannot be reproduced on demand (but is known to exist) is doubly more important than one that can be easily reproduced. They can occur via several scenarios, all very important: e.g. they might occur under conditions which may not be present in the lab, but very likely when the client is using the software; or they may occur as a result of a race condition.

When you have a bug that can be reproduced faithfully, maybe you can get away with filing it for now and dealing with it later. But if you have a "cannot reproduce always but definitely there" bug, you need to figure it out and fix it now, as the odd conditions that give rise to it now might become far more cryptic and infrequent further down the line.

One of the most famous cases of such a bug was the Therac-25. I'll let you read the extensive wikipedia article yourself, but essentially this was a radiotherapy machine which had just such a "race-condition" related bug. By the time it was spotted, and realised it was indeed a bug rather than "human error", many patients had died from accidental massive radiation overdoses. So yes, bugs can even kill, and "hard to reproduce" bugs even more so.

---

_{PS. Obviously, it does depend what the product is for and how the bug may be perceived to interfere (though this is hard to predict, and trivial bugs can eventually trigger wonderful cascades that become showstoppers). I'm not trying to over-dramatise anything here. But you never stated what the software is actually about, so I'm choosing to go with the "high risk software" assumption as a benchmark of importance.}

Answer 4

The proper reaction is not just "fail the test" and be done (or pass it and be done).

You should do some "due diligence" investigation into whether you can trigger the bug consistently. If you can find a consistent trigger, report that.

If you cannot find the root cause and trigger it consistently, report it as an intermittent bug and let management to decide if it is so obscure and inconsequential that it does not have to be fixed, or if more resources should be spent digging up the root cause.

QA is not to assure quality but to assist the management with providing information about the status of the application to make informed decisions when allocating resources.

Logging a trivial bug (which will not be fixed) has no business value so it is just a waste of time: yours (as a tester), and people who will triage the bug and decide to ignore it. (I discuss examples of what I consider "trivial" later).

When in doubt, enter the bug into the tracker.

Talk to the product owner about whether the bug is trivial (and should be ignored) or whether should be reported and fixed. Communication (in person) is cheap; don't communicate via the bug tracker.

In our company we have "product owners" - a person responsible for making decisions about their respective areas, and they are part of the triage team. So this is to give them a heads up, to avoid entering into the bug tracker something which they will reject during bug triage anyway, to avoid a waste of time.

It might make sense to track even bugs which should be ignored - so the decision to ignore them is recorded, and no more attempts to enter them are made. But your bug tracking system might not be the right place to enter them (might have too much overhead) - maybe some lighter resource, like a wiki page of known glitches (so other testers may learn what is a glitch and what is an honest bug needing to be reported). But for most situations, such a formal list of known glitches would be also too heavy-handed - it can be learned by communicating with other more senior testers.

Process is no replacement for communication.

A trivial bug is something really trivial: say, sometimes the page is not correctly redrawn when the window size changes, in a browser used by only a small proportion of your users, and can be solved with a page refresh, and has no consequences, and you cannot force it to happen consistently.

Another example of a trivial intermittent bug would be a traceback which occurred only once or very few times, where say some object was not properly instantiated, occurring at a time when there was some other outage/service disruption with a data provider, in code which had not changed for many weeks. Can you prove it is related to the disruption? Unlikely. Is it a good time investment to track down the root cause? Unlikely. So you don't enter it to bug tracker.

On some days we have several such tracebacks, and entering them into the bug tracker would make no business sense. The situation is different if they are numerous, and consistently appearing - those are well worth tracking and fixing.

Few commentators insist that every bug is worth reporting. Let me disagree. maybe we disagree what is a bug.

I am working in QA for few years, and found quite a few issues where we decided NOT to enter it to the bug tracker and NOT waste time investigating it, because we had bigger fish to fry. I am jealous if you have the time and resource to dive into each rabbit hole you find, but we don't.

Certainly we do have bugs entered years ago and still not resolved. Some are closed after few years, because parts of the system changed a lot since reporting, and we do not noticed it since.

But also certainly we do not enter every single one wayward traceback found in logs into bug tracker and spend hours investigating it.

Answer 5

50% is an extremely high rate. 500,000 of 1,000,000 customers will run into this bug. As a developer, I want this fixed, and I fell lucky that it is so easy to reproduce.

Obviously report "9 out of 20 times" with the bug report.

Answer 6

It should be reported that there is a bug and the test should be considered failing.

One of the details would be the frequency of failure.

Tests that fail 45% of the time are definitely failures for me. There is no hard and fast rule for % but when tests are in the 90%+ passing range - and usually over 95%, e.g. 19 passes in 20 runs I might consider the test to pass. However I will create or update another ticket to track those infrequent failures. 50% is a long way from that so would definitely be considered failing by me.

If this is something that your organization wishes to track you may want create / use a field in the ticket to indicate frequency of failure. Then you could report on it over time.

Answer 7

Intermittently reproducible bugs are just generally just as important as bugs that occur every time in my opinion. They are sometimes due to race conditions, which have a nasty habit of occurring more frequently when software moves from a test environment into a production environment.

Also, an intermittent bug that slips through into production has the potential to generate a lot of expensive support calls. Those will need to be paid for from someone's budget, although possibly not from the budget that is paying for the current development work.

I do both support and development on a mature (10 years old) commercial system, and about half the staff time spent on support calls is spent working around intermittent bugs that we've not yet figured out how to replicate. Knowing how to replicate those bugs could allow my company to save money.

Answer 8

• I had to face the same issue in past project.

• We have use of the snapshot or if more difficult then capture the video then the developer can easily check to steps for reproducing.

• If it issues for particular OS related then we have check into that os system.

For video capture have used below plugin:

• Screencastify - Screen Video Recorder

• Loom for Chrome

Answer 9

A lot depends on how your management feels about tracking issues that won't get resolved in the short term, and maybe not ever. This might not be a top priority for bug triaging, but do they want to build up information that a particular component appears to exhibit erratic behavior?

As a developer, I want our code to be as deterministic as possible. Having reproducible behavior helps a lot when tracking down more serious bugs. For example, if a particular code checkin caused a particular behavior to change in a long-running test case, one of the things I will try to do is dump out logs of intermediate activity and data structure values for before and after versions of the code, and compare them to see where the behavior diverges. It might be something as trivial as a minor change to a data structure that ultimately triggers bad behavior many minutes later. But if there are lots of erratic differences in the intermediate behavior, even if they don't ultimately matter, it can be nearly impossible to see where the significant behavior diverges.

Here's one example of code that can change the order in which items are processed. One thing we do a fair amount in our C++ code is to use STL sets to track if we have seen an object previously. So we might have a std::set<foo *> bar; containing the pointers to foo objects that we have seen. Now sometimes a programmer wants to iterate over the items in that set. What you don't want to do is iterate from bar.begin() to bar.end(), as you might over some other STL containers. The reason is that the items in the set will be ordered by the pointer value, so that another run of the code in which items are allocated elsewhere in memory could iterate in a different order. Algorithmically, it might not matter - either way you iterate over all the items. But if you want the intermediate results to be consistent, either you need to have the map sort by some other criteria, or you need to keep a parallel list or vector of seen pointers that you use for iteration. Sometimes little differences in ordering like this can have big macro effects on the code, and cause intermittent bugs.

Answer 10

Intermittent bugs are really bad and can break a company if the customers experience them (from my own experience.) Log the bug in the tracker as intermittent. It helps the company because it may be some time before someone spots a trend that points to the underlying problem. Further (different) testing may highlight other interesting code behaviour that helps to identify the reason for the first bug. It is important to check however, that the problem is not caused by your test. Try to rewrite it, reconsidering all possible inputs to the code. If you are sure and colleagues still seem uninterested or make fun, ask them if intermittent bugs can be caused by uninitialised variables. I am not saying that that is the cause of your bug but it should at least get your colleagues talking about the problem rather than ignoring the test result.

Answer 11

Speaking as a developer, I’d say it’s important to get to the bottom of it, right away. Being able to make it happen 50% of the time when it was not noticed before is much better than rare unpredictable anomalies. It might go back into hiding again on the next build.

Answer 12

Don't look for ways to interpret the phrasing of a test case to fit the observed outcome. The point is the product and whether it's fit to be used by its customers; your list of test cases is just a means to that end. This test case was created because your customers are expected to take whatever actions it entails, and the software should respond appropriately every time. If you pass it, you're telling the developers there's no problem in that part of the software when you know there is. Your co-workers with a "just pass it" attitude should be seeking opportunities in other fields.

As for why it only happens 50% of the time, that means that it's influenced by a factor that you haven't identified yet (and which also wasn't anticipated by the author of the test cases). I've seen issues like this that depended on some specific quirk of the test data, like the number of characters in a string or whether a particular character was present, or even whether this database record ID is higher or lower than that one. Log details of exactly what you did each time so a developer can retrace your steps if it gets assigned out to be fixed.

Answer 13

I would first cover other cases and set aside some time to re-visit it later.

On re-visit, I will investigate the scenario in a systematic way like peeling an onion.I will try to remove superfluous steps so that I will have the core steps to reproduce the issue in its barebone form.Once I reduce it to its core form then I will share my findings with the bug.

As per my experience in these situations, even if you cannot find the exact steps after investing lot of time but if you do it systematically at least your knowledge of the SUT will be improved significantly which will help you in one way or another in the testing process.

Answer 14

What should be reported when a bug only happens 50% of the time?

I've encountered this while on-the-job in QA and when I ask about it I find that my colleagues, when shown the test, do not seem very worried about this kind of behaviour. They seem happy to simply pass the test!

A bug that happens so often makes the program unusable or annoying or loses your work half the time, obviously, that isn't acceptable; but I'd be more concerned about the bigger problem, your colleagues.

A bug is always a bug, and in a computer program, it's undesirable. It's like seeing a flying insect fly in your window and outside again, when it is outside it's no longer a bug and can not reoccur ? - that is only likely if you keep the window closed.

What should I report? My instinct was to fail the test and go into a deeper dive around the issue.

In the case of a bug occurring so frequently you should attempt to determine when it occurs and when it does not. Document the severity of the effect. Is it a minor annoyance or does it cause data corruption that's hard to notice - priority is what determines which comes first and what resources (people and equipment) should be devoted to fixing it.

Since you have discovered it you are well advised to sufficiently document it. Because it occurs so often you should easily be able to accumulate sufficient information about its behaviour to make your own opinion and a report that someone else can read and make their own decision.

However, since this could be an issue of priority, maybe the Product Owner just wants to get other things done first.

Yes, well it depends on what it does, how often it occurs and what other work must be done; but ultimately if it can never be fixed and it ruins the program that's certainly an important factor too. There would be little point leaving a seemingly minor problem for last only to discover that without that particular problem being resolved the program couldn't be released.

But what's wrong with logging a trivial bug? What kind of reaction should I have to this kind of a situation?

If you see it you should report what you see. Even if it's only once. If it's a minor issue and you can clearly see that's the case then you still need to document it, use your belief that the issue is minor to justify not spending a lot of time on it.

Frequently things are not reproducible or are not seen by others, but for some people, the problem is clear as day. It can be a case of different hardware producing wildly varying results or people's perception of what constitutes a problem.

You don't want to be the one whom on release day says: "Fffft, I saw that bug weeks ago but didn't bother to mention it". Much better to say: "I saw it and documented it, but assigned it to a lower priority because ...".

No such thing as a little pregnant. A little used if it should be new. A little wrong when someone is paying for it to be right. If you reach a time or money limit you have documentation (maybe screenshots) that you can show the customer, if they sign-off on it you are covered, if they want it fixed you know what to fix (or to try to reproduce and declare it not reproducible or fixed).