19

If you find more bugs than you fix, then what should you do with the ever expanding bug queue?

The queue can grow for several reasons:

  • The bug is deemed too low priority to be worth fixing
  • The bug is edge case that has a low probability of happening
  • The business is prioritizing new functionality over fixing issues and is willing to accept this form of debt
  • You're either not finding or fixing bugs during development

It has been proposed in my team, that we close bugs older than x days, as they will get re-submitted if they're still an issue. Assuming you prioritize the higher severity bugs and try and fix known bugs in new changes, then this doesn't seem like too wild of an idea.

  • 3
    If you find more bugs than you're fixing then you might have bigger problems... – Wayne Werner Aug 21 '15 at 14:03
  • Just how old are you talking about? There's a huge difference between 30 day old bugs, and 3 year old bugs. As some of the other answers covered, having bug reports still in the system after 3 years probably means that you're right regarding just closing everything and cleaning out the system. It's dangerous, I think, to rely on "they will get resubmitted if they're still an issue" because you're likely relying on your customers to resubmit and this seems like a REALLY good way to piss off customers. – Rick Chatham Aug 24 '15 at 20:04
  • 1
    Obligatory link: jwz.org/doc/cadt.html – Nemo Dec 13 '15 at 7:48
  • See also sqa.stackexchange.com/a/17387/8992 – Michael Durrant Dec 11 '16 at 12:36

10 Answers 10

16

I would say that product owner (or whoever ultimately decides what the team works on) can decide that some bugs will not be fixed for one reason or another. Then I would close those bugs documenting this fact so that later it is clear what has happened.

I wouldn't close them just based on time as I see it should be conscious decision to close them and time itself isn't good enough indicator.

  • 1
    Besides agreeing with Edu's answer, I would also posit that you need to check if all of the issues you are finding are actually bugs or not. It might be that some of the edge case bugs are now non-issue bugs since the specs have changed. In the end, the person who is responsible for accepting the software from you is the person who can make that judgement call. They may decide to keep these issues in the backlog for awhile, in which case be prepared to go over this list again in the future. – Barry Hammer Aug 21 '15 at 13:30
  • 1
    +1 for documenting the close reason. Agree on a sensible bug-lifecycle. Possilbe close reasons besides "fixed" can include "invalid", "duplicate", "worksforme", and "wontfix" where we have the rule that when you resolve as "wontfix" you have to leave a comment, why it's a "wontfix" case. – Sumyrda - Reinstate Monica Aug 21 '15 at 15:32
  • It's also a good idea to document who made the call in case it turns out to be an unpopular decision. – David Cain Dec 5 '16 at 18:30
6

I would like to consider each of the cases separately:

  1. Low Priority - Low priority bugs do not stay low priority always. They always have a tendency to become high priority if enough of the users complain (or a important customer raises it). Closing doesn't seem to be a good option.

  2. Edge Cases - Remember Murphy's law. These tend to happen and they happen when you least expect it. Closing them is not an option.

  3. Business Priority - Even when the business priorities are different, bugs still exist. If the new version of software is green field, these bugs might not make sense - but still need to be verified.

  4. Not finding or Fixing bugs - Time to talk to the development team :)

As you can see, I am not for closing the bugs till they are fixed. What I suggest will be to use the priority and category fields in tracking tool and keep them hidden if needed. A bug entry takes time and effort - and it is waste of energy to close them without resolving them.

4

Bug is Bug , it does not matter how old it is. As a tester you should not close bug if it is not resolved yet by developer and verified by tester.

Close old bug without solve and insert it again that will just increase bug counts. It is fine that you concentrate on high priority bugs first but to close without resolve seems wrong.

But if that bug is from requirements which is removed or totally changed then tester can take approval from client in written that because of this requirement change , particular bug is meaningless and we should close. In this way you can close.

  • 1
    Agreed that it seems wrong but what is the purpose of keeping an old bug open if it's too obscure or low priority to ever be worked on. Why would the business decide, lets fix some really obscure bugs rather than invest in more commercially viable changes? A closed bug can still be referenced if needed. A growing bug queue also becomes too resource intensive to maintain, especially with a regularly changing code base. – Vincent Brazil Aug 21 '15 at 7:51
  • 1
    That can be because of improper management from development side , Continue enhancements to application, Work load, Critical release and a lot more. There could be many reason. – Helping Hands Aug 21 '15 at 7:55
3

In my experience, closing a bug simply because it's old is not a good idea. My organization used to do this as a policy, but it caused a number of problems.

  • The bug can look like it's fixed. Test teams would run queries to see which bugs were closed between two releases. If a bug was closed, that test case was no longer run (since the issue was presumed to be fixed) until the next major release cycle. This resulted in old, typically low-priority bugs frequently disappearing and re-appearing. To a manager, this can even look like your team keeps making the same mistake over and over again.
  • You lose potentially-valuable debugging information. A new bug will be filed if the issue is encountered again, but the new bug report is not connected to the original bug report. For bugs that are poorly understood or that only reproduce infrequently, you need all the information that you can get in order to debug it. It's usually more productive to add more information to an existing bug report than it is to file a second bug report for the same issue. For most bug tracking databases of a non-trivial size, it's not practical for a submitter to manually dig through all of the closed bugs for a possible match (it's hard enough getting them to look through the open bugs).
  • It can encourage people to ignore problems. Have a problem that's not mission-critical and that has a particularly difficult or unpopular solution? Just ignore it long enough and it will go away on its own. This is definitely not what you want to encourage your developers to do.
  • Hiding problems is almost always a bad thing. The only thing worse than having a hundred old bugs hanging around is to have a hundred bugs that you are no longer aware of. This will almost guarantee that they'll never get fixed, and won't prevent you from making the same mistakes again.

After a while, my group re-evaluated our policy and determined that it was making things worse, not better. We changed our policy so that instead of closing out old bugs, we had periodic meetings (after the end of each major project release/milestone) where we re-evaluated bugs that were over a certain age. This helped us understand why these old bugs existed in the first place, and why they've been able to stick around for so long. Many of them ended up being closed out, but for specific, meaningful reasons (e.g. the bug report didn't have enough information to start investigation, the test case was incorrect, or because the associated feature was cancelled and the issue is now moot) instead of simply because they were old. After a few rounds of this, our organization got better at identifying these sorts of bugs and changing our processes so that they didn't get filed in the first place. We were able to actually solve the problem of a large bug backlog instead of merely hiding it.

All this dances around the big issue, though. If you're creating bugs faster than you're fixing them, then you're headed for major problems. You really need to re-evaluate your development practices and make sure that you're improving the quality of your code base over time, not making it buggier. If you have that many bugs that are low-enough priority to ignore for an extended period of time, then it could also be a hint that your test plans are focusing on the wrong things.

2

I would not close bugs - you can "park" them - but it shouldn't look like the bugs are resolved. In our software release cycle we have ~2 months bug fixing phase - in this phase most developers are only fixing bugs no new features. This lets us get rid of most low priority bugs. And makes the product more "round".

In the end this helps you to ship a better product.

2

I think coming up with a good strategy of closing defects is very wise.

I once worked in a team which had over 3000+ major, minor and trivial bugs of which most were at-least 2-3 years old in the tracking system. At one point we wanted to start cleaning this and started fixing them. The result was hours and hours wasted on badly written defect reports that we could not reproduce either because it was already fixed and/or just not correctly described.

Just as never ending lists of features can be demotivating so will long lists of defects. You do not want to be alerted again and again for items you are not going to fix in the near future. Figure out what needs focus for the upcoming 2-3 months and plan accordingly.

Keep the blocking and critical defects at all times, but decide if you are going to fix the Major-Trivial defects and if not within a release-cycle time period archive them.

Personally I would try to get teams to go with a "Zero-defect" policy, do not create technical-debt and really understand how much new features (without defects) the teams can deliver.

2

When you close a bug, there should be a reason for closing the bug. The most obvious, of course, is fixed, but could not reproduce, not a bug and won't fix are equally valid.

Could Not Reproduce means exactly that - the person assigned to fix the bug has followed the step to reproduce the bug (your bug report does include those, right?) but cannot get the bug to occur on their development system. This status doesn't mean that the bug doesn't exist, merely that there are additional steps or conditions for the bug to occur; I have, for example, had bugs that occurred only under very specific temperature conditions. If this bug is important - see below - then it needs to go back to the testers to isolate the true conditions.

Not a Bug means that the behavior is not a bug, but that it is working as designed. You'll often see this when testing games - the tester believes something different should happen, but that is not what is in the design document. The tester might well be right, but testing is generally the wrong time for major design changes.

Will Not Fix is similar, but rather than working as design, it's a minor problem that won't greatly affect most users.

Keep in mind that not all bugs are created equal. Those which crash on start-up on every platform have the highest priority, while those which cause an insignificant graphical glitch under unusual circumstances would get a much lower priority. Programmers need to concentrate on fixing bugs based on their priority; as time goes by and the dreaded ship-date approaches, entire categories of bugs will be placed into the will not fix state. These bugs might be fixed in a later patch, but not right now.

So who assigns priority? Initially, the testers will, using some form of checklist or rubric. Next, the QA lead & programming lead should discuss new bugs before finalizing the categorization and assigning them to programmers to fix. The priority isn't fixed; additional testing might indicate the bugs are more or less serious, or occur more or less frequently. And, of course, as the delivery date approaches, some bugs seem far less important...

Once a bug has been resolved by the programmer, it still isn't closed. If it's been marked as fixed, then it has to go back to QA to verify that it has been fixed. Could not reproduce can either go back for more testing or left to sit if it's unimportant. Not a bug will typically need some confirmation that it's working as designed, while will not fix generally needs some sign off from upper management.

2

There are two issues in the question: how you use your bug tracking system, and how your deal with having too many bugs.

If you don't plan to fix a bug, you might as well make that apparent in your bug tracking system; these systems work best when they reflect reality. Any decent bug tracker will let you record whether the bug was closed because it was fixed or because you chose to ignore it.

There are lots of reasons why you might have more bugs than your developers can fix. Some reasons are actually software problems (e.g. developers writing too many bugs); some may not be (e.g. your testers are too picky, or lots of the bugs are duplicates, or lots of bugs are false positives.) Different organizations (at different times) have varying tolerances for bugs, too. If your product is too buggy, your users will let you know, and you'll realize something needs to change.

1

If you find more bugs than you fix, then what should you do with the ever expanding bug queue?

It has been proposed in my team, that we close bugs older than x days, as they will get re-submitted if they're still an issue.

It's a bad idea, depending on what "Close" means in your context.

If "close" means "we won't have to look at that bug any longer", then you have to assume the same issue will pop up next time you test significantly. Then, someone (the tester?) will have to decide "should I submit a new big report, reopen the old one, or just ignore the issue entirely". Each of these three alternatives has problems.

Seek a different status than "Closed". Perhaps "Deferred" or "Later" exist in your bug tracking system. Those statuses might mean "we know about the bug, aren't going to fix it in this release, but will re-triage for the next release". That would be my preference.

Then, spend some time on diagnosing why you have so many unfixed bugs, and work on a solution for that problem.

0

Despite being an edge case, or low priority, if you can still reproduce the reported bug, and the expected functionality of that system still dictates that it is a bug (instead of just an old requirement which is no longer valid), then it's still a bug, and should be left as so.

I wouldn't say its the QA's responsibility to maintain the product backlog (the product owner would do this) but in my opinion, a valid bug is still always a valid bug until it is fixed, and leaving these items in the backlog can provide good insight into the overall state of the software under test.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.