56

So, let's say I'm running some tests and run across some odd behavior that is a deviation from expected results, however, I quickly run the test again and the bug is not reproduced. I begin to question what's going on and run, say, 20 iterations of the test. I find that the unexpected behavior only occurs 9/20 times while the other 11/20 times the application runs smoothly.

I've encountered this while on-the-job in QA and when I ask about it I find that my colleagues, when shown the test, do not seem very worried about this kind of behavior. They seem happy to simply pass the test!

This has been IRKING me. What should I report? My instinct was to fail the test and go into a deeper dive around the issue. However, since this could be an issue of priority, maybe the Product Owner just wants to get other things done first. But what's wrong with logging a trivial bug? What kind of reaction should I have to this kind of a situation?

  • 45
    This kind of behavior indicates that the code is sensitive to external conditions, also called race-conditions. These kind of bugs are the hardest to identify and fix, for the exact reason that you describe, and should be taken very seriously because they may cause the system to fail for a customer in production in a non-reproducible way. You are lucky that it shows up as often as it does.I would suggest you talk to the Product Owner and underline that the "does not happen every time" is not a good, but a bad thing. – Thorbjørn Ravn Andersen Jun 30 '17 at 4:31
  • 12
    Just make sure this is a problem with software, and not with flaky tests. – charlie_pl Jun 30 '17 at 7:23
  • 21
    As a developer I'd be alarmed to see a bug that happens 50% of the time. It's a bug. – M Bailey Jun 30 '17 at 9:56
  • 2
    it really depends on what is being tested. If it's something uncontrollable, like measuring how many milliseconds a function takes to run, and the test env isn't the same as the production env, then it may be ignorable (or removed). Anything else and it's a bug, and likely a bad one. – James Jun 30 '17 at 13:15
  • 3
    I had a bug that failed about 10% of the time, and putting in any trace or debug code fixed it (nasty!) Needed perhaps 100 repeats to establish statistically whether it had gone not. Had to build custom hardware into the machine to debug it. Uncovered a whole class of race condition that had been ignored, and didn't show up in some other places. Any failure, investigate. Bugs come from you not fully understanding how the system works. If you haven't understood how the system works, you are building in catastrophic trouble for later. – Neil_UK Jul 1 '17 at 14:10

14 Answers 14

62

A bug that is only a bug 50% of the time is still a bug.

I would still report/document it as a regular bug, but with an "intermittent" label - letting to know both developers and management about the bug, and that it is not reproduced all the time.

This way the issue would get its priority by the management, and the mentioned people may generate some ideas about making the bug reproducible 100% of the time. You would also let this "IRKING" partially go away having the fact that you observed the issue recorded.

The important thing is to document as much information about the facts and observations when the bug is reproduced and when is not - record as much as possible - screenshots, logs, stacktrace, current theories and ideas. Think about things that vary from one test execution to another - test order, different pre- and post- conditions, network delays, user access control, etc.

  • 3
    Also, caching. Cronjobs. Timeouts. Date changes. Second hand marker (like a bug that only surfaces if the start and end of a job occur in different minutes, so if you start at :59 you see it but if you start at :07 you don't). More caching. Failure to cache. Should you blame caching? – Wildcard Jun 30 '17 at 20:03
  • 21
    @Wildcard, the inability to send an email more than 500 miles. – Mark Jun 30 '17 at 23:02
  • 1
    @Mark that was a fun read - have not yet encountered this story before. Thanks for posting the link! – alecxe Jun 30 '17 at 23:32
  • 2
    Not just "still a bug". Other things equal, a bug that only happens X % of the time is 100/X times as bad as one that always happens. – R.. Jul 2 '17 at 0:08
  • 1
    As a former developer, can you just give us a bug report that only shows up in the queue of things to work on 50% of the time? I think that'd be a lot more appropriate. – corsiKa Jul 4 '17 at 3:26
15

Try to get a reliably-failing test case, but at the end of the day, it's still a bug. Make sure to state that e.g. "hey, this only fails ~1/2 the time".

(Note that one way to dance around the issue if it's impolitic is to write a new test case that runs the test e.g. 100 times.)

I once reported an HW bug that occurred ~1-in-100k times on ~1/10 devices... (And I found it on iteration 9986/10k, and even that was only because I forgot to stop it before I left for the weekend...) The root cause was one of those "if the stars line up" bugs - a write in a 1 clock cycle window would tristate an output for a bit, and everything would be OK as long as said output retained its value for long enough.

  • 7
    It might be better to frame it as "hey, this only passes ~1/2 the time" instead, since that sounds more concerning than the other way. – Thunderforge Jul 1 '17 at 14:56
  • 1
    The Intel floating-point bug is another example. Only happened for a few values but it cost them $350 million in remediation, as it was never supposed to fail. – user207421 Jul 3 '17 at 1:48
  • Write a test that runs the 45%-failing test 20 times. New test fails. – kaay Jul 5 '17 at 6:31
11

A bug that cannot be reproduced on demand (but is known to exist) is doubly more important than one that can be easily reproduced. They can occur via several scenarios, all very important: e.g. they might occur under conditions which may not be present in the lab, but very likely when the client is using the software; or they may occur as a result of a race condition.

When you have a bug that can be reproduced faithfully, maybe you can get away with filing it for now and dealing with it later. But if you have a "cannot reproduce always but definitely there" bug, you need to figure it out and fix it now, as the odd conditions that give rise to it now might become far more cryptic and infrequent further down the line.

One of the most famous cases of such a bug was the Therac-25. I'll let you read the extensive wikipedia article yourself, but essentially this was a radiotherapy machine which had just such a "race-condition" related bug. By the time it was spotted, and realised it was indeed a bug rather than "human error", many patients had died from accidental massive radiation overdoses. So yes, bugs can even kill, and "hard to reproduce" bugs even more so.


PS. Obviously, it does depend what the product is for and how the bug may be perceived to interfere (though this is hard to predict, and trivial bugs can eventually trigger wonderful cascades that become showstoppers). I'm not trying to over-dramatise anything here. But you never stated what the software is actually about, so I'm choosing to go with the "high risk software" assumption as a benchmark of importance.

  • 2
    This is so true. If you know precisely what the undesirable behavior is and precisely in what circumstances it happens, your system is still predictable. If you don't know why something is happening, your entire system becomes unpredictable and logical reasoning about it goes out the window because you don't know which of your assumptions are false. From a falsehood anything can be proven. – Wildcard Jun 30 '17 at 20:10
  • 2
    A more rigorous treatment (pun intended) is in An Investigation of the Therac-25 Accidents (published in IEEE Computer). It is long (four long pages), but interesting, and it should be required reading for anyone involved in functional safety. – Peter Mortensen Jul 1 '17 at 13:58
8

The proper reaction is not just "fail the test" and be done (or pass it and be done).

You should do some "due diligence" investigation into whether you can trigger the bug consistently. If you can find a consistent trigger, report that.

If you cannot find the root cause and trigger it consistently, report it as an intermittent bug and let management to decide if it is so obscure and inconsequential that it does not have to be fixed, or if more resources should be spent digging up the root cause.

QA is not to assure quality but to assist the management with providing information about the status of the application to make informed decisions when allocating resources.

Logging a trivial bug (which will not be fixed) has no business value so it is just a waste of time: yours (as a tester), and people who will triage the bug and decide to ignore it. (I discuss examples of what I consider "trivial" later).

When in doubt, enter the bug into the tracker.

Talk to the product owner about whether the bug is trivial (and should be ignored) or whether should be reported and fixed. Communication (in person) is cheap; don't communicate via the bug tracker.

In our company we have "product owners" - a person responsible for making decisions about their respective areas, and they are part of the triage team. So this is to give them a heads up, to avoid entering into the bug tracker something which they will reject during bug triage anyway, to avoid a waste of time.

It might make sense to track even bugs which should be ignored - so the decision to ignore them is recorded, and no more attempts to enter them are made. But your bug tracking system might not be the right place to enter them (might have too much overhead) - maybe some lighter resource, like a wiki page of known glitches (so other testers may learn what is a glitch and what is an honest bug needing to be reported). But for most situations, such a formal list of known glitches would be also too heavy-handed - it can be learned by communicating with other more senior testers.

Process is no replacement for communication.

A trivial bug is something really trivial: say, sometimes the page is not correctly redrawn when the window size changes, in a browser used by only a small proportion of your users, and can be solved with a page refresh, and has no consequences, and you cannot force it to happen consistently.

Another example of a trivial intermittent bug would be a traceback which occurred only once or very few times, where say some object was not properly instantiated, occurring at a time when there was some other outage/service disruption with a data provider, in code which had not changed for many weeks. Can you prove it is related to the disruption? Unlikely. Is it a good time investment to track down the root cause? Unlikely. So you don't enter it to bug tracker.

On some days we have several such tracebacks, and entering them into the bug tracker would make no business sense. The situation is different if they are numerous, and consistently appearing - those are well worth tracking and fixing.

Few commentators insist that every bug is worth reporting. Let me disagree. maybe we disagree what is a bug.

I am working in QA for few years, and found quite a few issues where we decided NOT to enter it to the bug tracker and NOT waste time investigating it, because we had bigger fish to fry. I am jealous if you have the time and resource to dive into each rabbit hole you find, but we don't.

Certainly we do have bugs entered years ago and still not resolved. Some are closed after few years, because parts of the system changed a lot since reporting, and we do not noticed it since.

But also certainly we do not enter every single one wayward traceback found in logs into bug tracker and spend hours investigating it.

  • 14
    "Logging a trivial bug (which will not be fixed) has no business value" - as a developer, completely disagree. Almost all truly major problems are a collection of "trivial" problems, though not all trivial problems will be part of a major problem down the line. If untracked, major problems will be much harder to fix than if the issue was tracked symptoms/potential causes identified, which you need to do to identify that it's a trivial bug anyway. Management should absolutely be able to decide it's not worth fixing, but it's worth tracking. – Phoshi Jun 30 '17 at 9:44
  • 4
    So rather than log the bug so that it can be discussed/prioritised during the triage session when all the relevant people are in place and ready to triage, you advocate going around the office interrupting numerous people individually to talk about some potentially trivial issue where you'll have to explain it n-times to each user and try to collate each person's opinion and act as a go-between? And you think that's more efficient "communication". No. Record the bug (with detail), move on. Let the triage team make the decision during the triage session, that's their job. – freedomn-m Jun 30 '17 at 11:52
  • 11
    I'm sorry, but the first half of this answer is great, but the second half is just wrong... An intermittent bug is still a bug and needs to be reported. What might seem trivial to one person, may be a showstopper to another. While I agree that communication is paramount, all issues MUST be logged or you lose visibility, and it will eventually increase cycle time and reduce overall quality – Taegost Jun 30 '17 at 12:58
  • 2
    @freedomn-m - Not interrupting numerous people. In our organization, we have "customer representatives" responsible to make decisions about they respective areas, and they are part of the triage team. So this is not interrupting them, but giving them heads up. If they are going to say at the triage that bug is trivial and not worth fixing, why waste time? – Peter M. Jun 30 '17 at 14:55
  • 1
    @Phoshi - I added example of a bug (traceback) which is not worth tracking in. I am former developer, working in QA now in automated regression testing and part of my daily routine is analyzing tracebacks, detecting actionable ones and ignoring trivial. – Peter M. Jun 30 '17 at 15:11
6

50% is an extremely high rate. 500,000 of 1,000,000 customers will run into this bug. As a developer, I want this fixed, and I fell lucky that it is so easy to reproduce.

Obviously report "9 out of 20 times" with the bug report.

  • That's not what 50% means here. A test is failing 50% of the time, but the test might be testing code that is used by 1 in a million of users. – Davor Jun 30 '17 at 9:51
  • 2
    @Davor which, of course, depending on one's perspective, potentially makes matters worse rather than less worthy of requiring attention. – Tasos Papastylianou Jun 30 '17 at 17:27
  • 2
    @Davor Or code that is used twice daily by all users, in which case they will all encounter it every day. – user207421 Jul 3 '17 at 1:47
  • 1
    @EJP - exactly. Test failing 50% of the time tells us literally nothing about how often this bug is encountered in production. It may be on every login, it may be never. – Davor Jul 3 '17 at 11:11
4

It should be reported that there is a bug and the test should be considered failing.

One of the details would be the frequency of failure.

Tests that fail 45% of the time are definitely failures for me. There is no hard and fast rule for % but when tests are in the 90%+ passing range - and usually over 95%, e.g. 19 passes in 20 runs I might consider the test to pass. However I will create or update another ticket to track those infrequent failures. 50% is a long way from that so would definitely be considered failing by me.

If this is something that your organization wishes to track you may want create / use a field in the ticket to indicate frequency of failure. Then you could report on it over time.

  • 8
    Tests should give the same result every time. If they don't something is broken. Badly. – Thorbjørn Ravn Andersen Jun 30 '17 at 4:32
  • 1
    A test that fail once in 1 million runs or an error that has never reoccurred may be due to exceptional unusual circumstances that never reoccur or a dependency or configuration issue that does not reoccur. So not automatically "Badly" for those. – Michael Durrant Jun 30 '17 at 10:25
  • 7
    You cannot tell until you have understood completely why it failed, and what the fix was. A once in a million failure may be a rarely triggered race condition. – Thorbjørn Ravn Andersen Jun 30 '17 at 12:25
  • 1
    @ThorbjørnRavnAndersen While that is true, the likelihood of failing should be taken into account when it comes to assigning bug fixing priority. (Well, that and the bug severity. If your 1-in-a-million bug erases the database, that sounds like it's still high-prio.) – Llewellyn Jul 1 '17 at 21:20
  • 1
    @hiergiltdiestfu Yes, in those cases even 99.99% may not be enough as 1 death in 10,000 is not acceptable. Some products should have 99.9999999% – Michael Durrant Jul 3 '17 at 11:38
4

Intermittently reproducible bugs are just generally just as important as bugs that occur every time in my opinion. They are sometimes due to race conditions, which have a nasty habit of occurring more frequently when software moves from a test environment into a production environment.

Also, an intermittent bug that slips through into production has the potential to generate a lot of expensive support calls. Those will need to be paid for from someone's budget, although possibly not from the budget that is paying for the current development work.

I do both support and development on a mature (10 years old) commercial system, and about half the staff time spent on support calls is spent working around intermittent bugs that we've not yet figured out how to replicate. Knowing how to replicate those bugs could allow my company to save money.

  • "as important" -> "MORE important." Which the rest of your answer bears out. – Wildcard Jun 30 '17 at 23:32
3

A lot depends on how your management feels about tracking issues that won't get resolved in the short term, and maybe not ever. This might not be a top priority for bug triaging, but do they want to build up information that a particular component appears to exhibit erratic behavior?

As a developer, I want our code to be as deterministic as possible. Having reproducible behavior helps a lot when tracking down more serious bugs. For example, if a particular code checkin caused a particular behavior to change in a long-running test case, one of the things I will try to do is dump out logs of intermediate activity and data structure values for before and after versions of the code, and compare them to see where the behavior diverges. It might be something as trivial as a minor change to a data structure that ultimately triggers bad behavior many minutes later. But if there are lots of erratic differences in the intermediate behavior, even if they don't ultimately matter, it can be nearly impossible to see where the significant behavior diverges.

Here's one example of code that can change the order in which items are processed. One thing we do a fair amount in our C++ code is to use STL sets to track if we have seen an object previously. So we might have a std::set<foo *> bar; containing the pointers to foo objects that we have seen. Now sometimes a programmer wants to iterate over the items in that set. What you don't want to do is iterate from bar.begin() to bar.end(), as you might over some other STL containers. The reason is that the items in the set will be ordered by the pointer value, so that another run of the code in which items are allocated elsewhere in memory could iterate in a different order. Algorithmically, it might not matter - either way you iterate over all the items. But if you want the intermediate results to be consistent, either you need to have the map sort by some other criteria, or you need to keep a parallel list or vector of seen pointers that you use for iteration. Sometimes little differences in ordering like this can have big macro effects on the code, and cause intermittent bugs.

3
  • I had to face the same issue in past project.

  • We have use of the snapshot or if more difficult then capture the video then the developer can easily check to steps for reproducing.

  • If it issues for particular OS related then we have check into that os system.

For video capture have used below plugin:

https://chrome.google.com/webstore/detail/screencastify-screen-vide/mmeijimgabbpbgpdklnllpncmdofkcpn?hl=en

https://chrome.google.com/webstore/detail/loom-video-recorder-scree/liecbddmkiiihnedobmlmillhodjkdmb

2

Intermittent bugs are really bad and can break a company if the customers experience them (from my own experience.) Log the bug in the tracker as intermittent. It helps the company because it may be some time before someone spots a trend that points to the underlying problem. Further (different) testing may highlight other interesting code behaviour that helps to identify the reason for the first bug. It is important to check however, that the problem is not caused by your test. Try to rewrite it, reconsidering all possible inputs to the code. If you are sure and colleagues still seem uninterested or make fun, ask them if intermittent bugs can be caused by uninitialised variables. I am not saying that that is the cause of your bug but it should at least get your colleagues talking about the problem rather than ignoring the test result.

1

Speaking as a developer, I’d say it’s important to get to the bottom of it, right away. Being able to make it happen 50% of the time when it was not noticed before is much better than rare unpredictable anomalies. It might go back into hiding again on the next build.

1

Don't look for ways to interpret the phrasing of a test case to fit the observed outcome. The point is the product and whether it's fit to be used by its customers; your list of test cases is just a means to that end. This test case was created because your customers are expected to take whatever actions it entails, and the software should respond appropriately every time. If you pass it, you're telling the developers there's no problem in that part of the software when you know there is. Your co-workers with a "just pass it" attitude should be seeking opportunities in other fields.

As for why it only happens 50% of the time, that means that it's influenced by a factor that you haven't identified yet (and which also wasn't anticipated by the author of the test cases). I've seen issues like this that depended on some specific quirk of the test data, like the number of characters in a string or whether a particular character was present, or even whether this database record ID is higher or lower than that one. Log details of exactly what you did each time so a developer can retrace your steps if it gets assigned out to be fixed.

0

I would first cover other cases and set aside some time to re-visit it later.

On re-visit, I will investigate the scenario in a systematic way like peeling an onion.I will try to remove superfluous steps so that I will have the core steps to reproduce the issue in its barebone form.Once I reduce it to its core form then I will share my findings with the bug.

As per my experience in these situations, even if you cannot find the exact steps after investing lot of time but if you do it systematically at least your knowledge of the SUT will be improved significantly which will help you in one way or another in the testing process.

0

What should be reported when a bug only happens 50% of the time?

I've encountered this while on-the-job in QA and when I ask about it I find that my colleagues, when shown the test, do not seem very worried about this kind of behaviour. They seem happy to simply pass the test!

A bug that happens so often makes the program unusable or annoying or loses your work half the time, obviously, that isn't acceptable; but I'd be more concerned about the bigger problem, your colleagues.

A bug is always a bug, and in a computer program, it's undesirable. It's like seeing a flying insect fly in your window and outside again, when it is outside it's no longer a bug and can not reoccur ? - that is only likely if you keep the window closed.

What should I report? My instinct was to fail the test and go into a deeper dive around the issue.

In the case of a bug occurring so frequently you should attempt to determine when it occurs and when it does not. Document the severity of the effect. Is it a minor annoyance or does it cause data corruption that's hard to notice - priority is what determines which comes first and what resources (people and equipment) should be devoted to fixing it.

Since you have discovered it you are well advised to sufficiently document it. Because it occurs so often you should easily be able to accumulate sufficient information about its behaviour to make your own opinion and a report that someone else can read and make their own decision.

However, since this could be an issue of priority, maybe the Product Owner just wants to get other things done first.

Yes, well it depends on what it does, how often it occurs and what other work must be done; but ultimately if it can never be fixed and it ruins the program that's certainly an important factor too. There would be little point leaving a seemingly minor problem for last only to discover that without that particular problem being resolved the program couldn't be released.

But what's wrong with logging a trivial bug? What kind of reaction should I have to this kind of a situation?

If you see it you should report what you see. Even if it's only once. If it's a minor issue and you can clearly see that's the case then you still need to document it, use your belief that the issue is minor to justify not spending a lot of time on it.

Frequently things are not reproducible or are not seen by others, but for some people, the problem is clear as day. It can be a case of different hardware producing wildly varying results or people's perception of what constitutes a problem.

You don't want to be the one whom on release day says: "Fffft, I saw that bug weeks ago but didn't bother to mention it". Much better to say: "I saw it and documented it, but assigned it to a lower priority because ...".

No such thing as a little pregnant. A little used if it should be new. A little wrong when someone is paying for it to be right. If you reach a time or money limit you have documentation (maybe screenshots) that you can show the customer, if they sign-off on it you are covered, if they want it fixed you know what to fix (or to try to reproduce and declare it not reproducible or fixed).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.