What if you’ve found a minor bug, affecting 1% of users and your actions?
asked Sep 26, 2019 at 7:29
-
46What is your normal action for defects, and why would it differ now?– Niels van ReijmersdalCommented Sep 26, 2019 at 8:42
-
21Impact is part of the same severity measure as how frequently it occurs. You might have a bug that affects everyone that crashes your program once a day, or a bug that will only affect 0.00001% of all your users, but launches a nuke when it happens... It's the combination of both that determines what you need to do to resolve it (or not)– BaldrickkCommented Sep 26, 2019 at 16:53
-
3Are you asking whether you should veto shipping the product (a power which an independent QA may have)? Is your product safety relevant (automotive, rail, power etc.)? Is it easy or hard to update the product in the field (assuming it's a software error)? When we wrote software for car radios it had at a minimum to meet the following two requirements: (1) Not drain the car battery. (2) Be updatable in the field. (In particular, it was not necessary to receive radio or produce sounds!)– Peter - Reinstate MonicaCommented Sep 27, 2019 at 13:13
-
3How many is 1% of your users? 100 users? 100,000 users?– SchwernCommented Sep 28, 2019 at 1:31
-
7I’m shocked that this question got upvotes.– RyanCommented Sep 28, 2019 at 3:35
|
Show 7 more comments
10 Answers
What I would do with any other bug - report it, write the bug report.
-
26I disagree. It was asked what would I do if I found a minor bug. I provided an answer. You might not like it, but it is an answer (and a correct one). Commented Sep 26, 2019 at 9:44
-
31Upvote this one. Agree with Mate. I do not see any critisism here. Qa engineer should report the defect whatever severity it has. The task of management is to triage and prioritize. Commented Sep 26, 2019 at 9:54
-
11The answer could be expanded on, but yes, it's a perfectly valid answer and sums up what I'd do as well. Commented Sep 26, 2019 at 12:38
-
7Totally agree that it's a valid answer. Unless the OP is not just in QA, but managing the priorities of bug fixing in development, reporting is entirely what their duty is. Commented Sep 26, 2019 at 22:52
-
3Not to mention the job security the defect report promotes... Why would someone not report a defect when it is part of their job description?– user41726Commented Sep 27, 2019 at 8:43
1% of 100 users is a very different issue to 1% of 1,000,000 users - make your team and stakeholders aware of the issue (preferably in writing, with a defect report) and then they can make the decision on the priority / severity of the issue.
It might be a minor issue for you, but a huge issue for the company.
-
15It also depends on how much your 100 users pay for the software. Crytek3 is $1.2 Million (2019), you can be sure that the user affected by the bug in your example will want it fixed. You should use your company's bug tracking system and report the bug, as with any other bug. Commented Sep 26, 2019 at 23:01
-
1@TomEberhard I'm not sure whether the price is really relevant. If I buy a $5 item on e-bay and it doesn't work, I tend to do the same thing I'd do for a $1000 item: complain until I get it fixed or refunded. Commented Sep 27, 2019 at 13:00
-
4@DmitryGrigoryev Price is very relevant: For a $5 item that doesn't work, I curse my luck and buy a similar item from a different vendor and hope it works this time. For a $1000 item, I complain till fixed if that's an option, or return it for a refund. Commented Sep 27, 2019 at 19:14
-
2On of my favorate quotes is from the author of the linux (and thus android) copy-on-write kernal functionality: "This code is running on over 7 billion devices, A once in a million year bug, will be hit 20 times every day." Commented Sep 28, 2019 at 22:15
This is called Risk Analysis.
By the book, the over-simplified step is to analyze Impact x Frequency. Things that happen rarely but with huge impact can be prioritized, as well as things with little impact but too frequent.
For a deeper understanding, I would suggest watching Michael Bolton's talk on Risk Analysis. There he questions our biases that may induce you to shallow analysis and suggests how to tell a Risk Story, so you can make stakeholders aware of the risks:
answered Sep 26, 2019 at 9:02
-
3Risk Analysis in the context of testing, and as far as I understand also for Michael Bolton, is about designing and prioritizing tests and about reporting of problems. It is not about deciding on what to do next, that's something product managers are responsible to– RsfCommented Sep 26, 2019 at 10:11
-
2The question asked "What QA should do?" The risk analysis, resulting in a Risk Story is my suggestion. Commented Sep 26, 2019 at 10:24
-
1Got it, although I have never heard anyone use this term or present it this way- it should simply be an integral part of (any?) bug report– RsfCommented Sep 26, 2019 at 10:28
There's already 2 correct answers but I can't stress this enough.
You found a bug, you file a bug report.
It doesn't matter who it affects or how. It could conceivably affect 0 real users and still be a bug and you still file a bug report.
A QA's job is not to determine whether or how quickly bugs are fixed. A QA's job is to find bugs and make it known that bugs were found.
-
5It's also important to file the bug so that it ends up in the internal documentation (things we'll fix when we have time), the external documentation (known bugs!) and can be allotted resources when available. A bug that's "flying under the radar" will be forgotten and will not receive the attention it should (even if that's very little). A bug tracking system must have a way to mark bugs as "postponed", meaning they do not show in daily meetings etc. but are tracked anyway. Commented Sep 27, 2019 at 13:12
-
3@PeterA.Schneider And important that it stays in documentation until it's properly investigated, even if the original reporter has gotten tired of waiting and stopped using the product. Commented Sep 27, 2019 at 16:24
-
4Also the next QA doesn't have to spend time and effort on documenting the same bug– xyiousCommented Sep 27, 2019 at 19:12
-
3As a developer - yes! Even if it doesn't necessarily affect a real user I want to know about it because it could easily show me a problem area in the code that might blow up in the future.– pipeCommented Sep 28, 2019 at 8:27
There are plenty of answers here talking about how all users matter. Essentially, this is a good reason to care about rare bugs, but I thought I would give a developer's perspective on some other reasons.
First, and most importantly, bugs reveal a mistake in my thinking. It is important to correct my thinking so that I can correct the rest of my code. I once had a bug where a progress bar didn't disappear on time for a small minority of cases. As the dev leading that part of the product, I recognised it as an indication that the clean up routine for the whole system was running just a tad later than I thought it was, and after things that it needed to run before. Occasional ui glitches may seem a minor bug, but the issue it revealed was an absolute time bomb.
Second, and important but less so, I am mathematical enough to know 1 percent is not that small a number. I am a very competent bug writer. All devs are. I have written hundreds of the things, and that is only the ones I am aware of! So, if there were a policy that only bugs affecting over 1 percent get flagged and fixed, chances are almost all users would wind up with a bug that doesn't go away.
Third, even things that do not immediately justify fixing are worth knowing. If it turns out that this bug is really hard to resolve and not immediately urgent, I may mark it wontfix. Even so, when I get chance to do a chunky rewrite for other reasons, I would keep a checklist of defects and avoid them. But if I don't know, I can't design for it.
It really depends on many different things like:
- What is the severity of this bug?
- What is the effort of fixing it?
- 1% of how many users?
There are a lot more things to consider, but a bug is a bug. It needs to be reported and the decision should be taken by the managers. So, I would simply report it.
-
3None of your bullet points relates to what the QA engineer should do. They aren't relevant to your last paragraph.– chepnerCommented Sep 27, 2019 at 15:17
How do you know bug will affect only 1% of the Users?
Even if bug severity is low (impact low number of user as you say), business like Product owners might increase bug priority and fix it. If you do not create bug report for it, they are missing important information about the SUT.
You should always report bug, once you find it.
-
1It affects 1% of users, since the bug could appear whenever the last two digits of the user ID are 42.– JensCommented Sep 27, 2019 at 16:04
-
I seemed to recall if it was when the ID ended on 24. Commented Sep 29, 2019 at 6:14
A bug that is considered minor today may become critical tomorrow. As an example, many external HDDs only implemented USB mass storage properly, and had buggy USB attached SCSI implementation. That was a minor issue until UAS driver made it to the Linux kernel, resulting in data loss or inability to use such drives with Linux PCs after an update.
Having a minor bug in the bug tracker that stays unfixed is better than finding a bug, not reporting it and forgetting about it until the day it strikes back and you have to search for the root cause all over again.
What I'd do is, as many already said here, to write a bug report. In it you describe what impact it has (not only in terms of affected users but what it causes as well) and with what frequency it happens.
As soon as it's described, you and your team can decide what to do next. Perhaps the business representatives, or the team, decide what priority this bug has. Then you (most likely the developers) can either start working on it, or it will be left for later.
There're some other alternative paths which I don't recommend:
- you don't file a bug report - you're running the risk it is actually a serious issue and you've just failed to find all impacts of it; in such a case, it'll backfire on you later
- you talk about it with the rest of your team first - still better than 1. since others might have important inputs on the topic, but you're still running the risk of e.g. forgetting it after the discussion
...and we'd probably be able to come up with more, these seem to be the most common ones.
answered Sep 26, 2019 at 22:20
I would suggest that you bring it to the client/sponsor's attention and let them prioritize the fix. At the end of the day it's their call as to the severity of the issue.
