25

What if you’ve found a minor bug, affecting 1% of users and your actions?

  • 45
    What is your normal action for defects, and why would it differ now? – Niels van Reijmersdal Sep 26 at 8:42
  • 21
    Impact is part of the same severity measure as how frequently it occurs. You might have a bug that affects everyone that crashes your program once a day, or a bug that will only affect 0.00001% of all your users, but launches a nuke when it happens... It's the combination of both that determines what you need to do to resolve it (or not) – Baldrickk Sep 26 at 16:53
  • 3
    Are you asking whether you should veto shipping the product (a power which an independent QA may have)? Is your product safety relevant (automotive, rail, power etc.)? Is it easy or hard to update the product in the field (assuming it's a software error)? When we wrote software for car radios it had at a minimum to meet the following two requirements: (1) Not drain the car battery. (2) Be updatable in the field. (In particular, it was not necessary to receive radio or produce sounds!) – Peter - Reinstate Monica Sep 27 at 13:13
  • 3
    How many is 1% of your users? 100 users? 100,000 users? – Schwern Sep 28 at 1:31
  • 5
    I’m shocked that this question got upvotes. – Ryan Sep 28 at 3:35

10 Answers 10

89

What I would do with any other bug - report it, write the bug report.

  • 26
    I disagree. It was asked what would I do if I found a minor bug. I provided an answer. You might not like it, but it is an answer (and a correct one). – Mate Mrše Sep 26 at 9:44
  • 31
    Upvote this one. Agree with Mate. I do not see any critisism here. Qa engineer should report the defect whatever severity it has. The task of management is to triage and prioritize. – Alexey R. Sep 26 at 9:54
  • 11
    The answer could be expanded on, but yes, it's a perfectly valid answer and sums up what I'd do as well. – Kate Paulk Sep 26 at 12:38
  • 7
    Totally agree that it's a valid answer. Unless the OP is not just in QA, but managing the priorities of bug fixing in development, reporting is entirely what their duty is. – George M Reinstate Monica Sep 26 at 22:52
  • 3
    Not to mention the job security the defect report promotes... Why would someone not report a defect when it is part of their job description? – user41726 Sep 27 at 8:43
49

1% of 100 users is a very different issue to 1% of 1,000,000 users - make your team and stakeholders aware of the issue (preferably in writing, with a defect report) and then they can make the decision on the priority / severity of the issue.

It might be a minor issue for you, but a huge issue for the company.

  • 15
    It also depends on how much your 100 users pay for the software. Crytek3 is $1.2 Million (2019), you can be sure that the user affected by the bug in your example will want it fixed. You should use your company's bug tracking system and report the bug, as with any other bug. – TomEberhard Sep 26 at 23:01
  • 1
    @TomEberhard I'm not sure whether the price is really relevant. If I buy a $5 item on e-bay and it doesn't work, I tend to do the same thing I'd do for a $1000 item: complain until I get it fixed or refunded. – Dmitry Grigoryev Sep 27 at 13:00
  • 4
    @DmitryGrigoryev Price is very relevant: For a $5 item that doesn't work, I curse my luck and buy a similar item from a different vendor and hope it works this time. For a $1000 item, I complain till fixed if that's an option, or return it for a refund. – TomEberhard Sep 27 at 19:14
  • 1
    On of my favorate quotes is from the author of the linux (and thus android) copy-on-write kernal functionality: "This code is running on over 7 billion devices, A once in a million year bug, will be hit 20 times every day." – Lyndon White Sep 28 at 22:15
29

This is called Risk Analysis.

By the book, the over-simplified step is to analyze Impact x Frequency. Things that happen rarely but with huge impact can be prioritized, as well as things with little impact but too frequent.

For a deeper understanding, I would suggest watching Michael Bolton's talk on Risk Analysis. There he questions our biases that may induce you to shallow analysis and suggests how to tell a Risk Story, so you can make stakeholders aware of the risks:

enter image description here

  • 3
    Risk Analysis in the context of testing, and as far as I understand also for Michael Bolton, is about designing and prioritizing tests and about reporting of problems. It is not about deciding on what to do next, that's something product managers are responsible to – Rsf Sep 26 at 10:11
  • 2
    The question asked "What QA should do?" The risk analysis, resulting in a Risk Story is my suggestion. – João Farias Sep 26 at 10:24
  • 1
    Got it, although I have never heard anyone use this term or present it this way- it should simply be an integral part of (any?) bug report – Rsf Sep 26 at 10:28
25

There's already 2 correct answers but I can't stress this enough.

You found a bug, you file a bug report.

It doesn't matter who it affects or how. It could conceivably affect 0 real users and still be a bug and you still file a bug report.

A QA's job is not to determine whether or how quickly bugs are fixed. A QA's job is to find bugs and make it known that bugs were found.

  • 5
    It's also important to file the bug so that it ends up in the internal documentation (things we'll fix when we have time), the external documentation (known bugs!) and can be allotted resources when available. A bug that's "flying under the radar" will be forgotten and will not receive the attention it should (even if that's very little). A bug tracking system must have a way to mark bugs as "postponed", meaning they do not show in daily meetings etc. but are tracked anyway. – Peter - Reinstate Monica Sep 27 at 13:12
  • 3
    @PeterA.Schneider And important that it stays in documentation until it's properly investigated, even if the original reporter has gotten tired of waiting and stopped using the product. – No U Sep 27 at 16:24
  • 4
    Also the next QA doesn't have to spend time and effort on documenting the same bug – xyious Sep 27 at 19:12
  • 3
    As a developer - yes! Even if it doesn't necessarily affect a real user I want to know about it because it could easily show me a problem area in the code that might blow up in the future. – pipe Sep 28 at 8:27
7

There are plenty of answers here talking about how all users matter. Essentially, this is a good reason to care about rare bugs, but I thought I would give a developer's perspective on some other reasons.

First, and most importantly, bugs reveal a mistake in my thinking. It is important to correct my thinking so that I can correct the rest of my code. I once had a bug where a progress bar didn't disappear on time for a small minority of cases. As the dev leading that part of the product, I recognised it as an indication that the clean up routine for the whole system was running just a tad later than I thought it was, and after things that it needed to run before. Occasional ui glitches may seem a minor bug, but the issue it revealed was an absolute time bomb.

Second, and important but less so, I am mathematical enough to know 1 percent is not that small a number. I am a very competent bug writer. All devs are. I have written hundreds of the things, and that is only the ones I am aware of! So, if there were a policy that only bugs affecting over 1 percent get flagged and fixed, chances are almost all users would wind up with a bug that doesn't go away.

Third, even things that do not immediately justify fixing are worth knowing. If it turns out that this bug is really hard to resolve and not immediately urgent, I may mark it wontfix. Even so, when I get chance to do a chunky rewrite for other reasons, I would keep a checklist of defects and avoid them. But if I don't know, I can't design for it.

5

It really depends on many different things like:

  • What is the severity of this bug?
  • What is the effort of fixing it?
  • 1% of how many users?

There are a lot more things to consider, but a bug is a bug. It needs to be reported and the decision should be taken by the managers. So, I would simply report it.

  • 3
    None of your bullet points relates to what the QA engineer should do. They aren't relevant to your last paragraph. – chepner Sep 27 at 15:17
5

How do you know bug will affect only 1% of the Users?

Even if bug severity is low (impact low number of user as you say), business like Product owners might increase bug priority and fix it. If you do not create bug report for it, they are missing important information about the SUT.

You should always report bug, once you find it.

  • 1
    It affects 1% of users, since the bug could appear whenever the last two digits of the user ID are 42. – Jens Sep 27 at 16:04
  • I seemed to recall if it was when the ID ended on 24. – Michael Karas Sep 29 at 6:14
5

A bug that is considered minor today may become critical tomorrow. As an example, many external HDDs only implemented USB mass storage properly, and had buggy USB attached SCSI implementation. That was a minor issue until UAS driver made it to the Linux kernel, resulting in data loss or inability to use such drives with Linux PCs after an update.

Having a minor bug in the bug tracker that stays unfixed is better than finding a bug, not reporting it and forgetting about it until the day it strikes back and you have to search for the root cause all over again.

3

What I'd do is, as many already said here, to write a bug report. In it you describe what impact it has (not only in terms of affected users but what it causes as well) and with what frequency it happens.

As soon as it's described, you and your team can decide what to do next. Perhaps the business representatives, or the team, decide what priority this bug has. Then you (most likely the developers) can either start working on it, or it will be left for later.

There're some other alternative paths which I don't recommend:

  1. you don't file a bug report - you're running the risk it is actually a serious issue and you've just failed to find all impacts of it; in such a case, it'll backfire on you later
  2. you talk about it with the rest of your team first - still better than 1. since others might have important inputs on the topic, but you're still running the risk of e.g. forgetting it after the discussion

...and we'd probably be able to come up with more, these seem to be the most common ones.

2

I would suggest that you bring it to the client/sponsor's attention and let them prioritize the fix. At the end of the day it's their call as to the severity of the issue.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.