What would be the strategy when too many bugs are found in the final round of testing?

Question

The QA team starts testing a software/product and there are “way too many” defects…!
Every other scenario is failing, new flows are explored & clarifications sought.
What would be the strategy now?
It’s a complex application with tight schedule. Too many defects add cherry on the cake. The blame game starts. But at the end, development & QA are expected to work as a team & deliver the product/software.
What are the options?

• Reject the build. But that would mean delay in delivery.
• Get into a war room, daily. It helps to triage the defects & increase the seriousness.

What are the positive approaches at this condition?

Answer 1

It depends

I've been in this situation more times than I care to remember, and the general process I use works this way:

1. Triage all findings - At this stage I would be working to triage all bug findings into 4 classes:
   • Blockers - any bug that makes it impossible for the product to perform the basic user acceptance tests, or breaks crucial functionality. The kind of bugs that fall into this category are things like being unable to use the feature that's being developed, the application crashes each time you try to access the new feature, you can't actually purchase anything in a webstore... the kind of bug the completely ruins the software.
   • Critical - any other bug that has a high risk of occurring live and a high impact. Inaccurate financial calculations fall here, as do major security flaws. They don't stop users, but they have serious negative impacts.
   • Moderate - Bugs that could occur and would cause embarrassing and/or expensive consequences if they happen, but which don't break any laws or cause serious, blocking problems.
   • Minor - Anything that doesn't fall into the other categories - these are the bugs that can wait.
2. Report Triaged Findings - this is where things can get challenging. Once you and your team agree on the status of the bugs you are reporting, you need to provide your superiors with a short summary of your findings. The format I prefer is something along these lines:
   • Number of Blockers - The number of blocking issues currently not fixed. I'll often give this number in bold red text.
   • Number of Critical Bugs - again, this is the count of bugs that haven't been fixed.
   • State of Release - for this I generally use a three color system. Red means in my view there is no way the targeted deadline can be met without releasing a blocker. Yellow means that the deadline is possible but at risk and there may be critical bugs released. Green means that everything is on target to release without any blockers or critical bugs. I'll typically start sending this once a week as soon as I'm aware a targeted release is at risk. In the last week to 2 weeks before release (depending on how bad things are), I'll send the message daily.
3. Escalate Findings - the first recipient of my triaged findings is generally my manager. If I don't see any attempt to fix the blockers, I'll start including my manager's manager. The timeline for escalation depends on the timeline to release and how bad things are. I have escalated all the way to the company CEO, which ultimately forced that particular planned release to become a beta (there were contractual requirements).

Some things that are worth remembering:

• We don't have go/no-go authority. There may be contracted requirements to release on a particular date.
• We are not gatekeepers. Our role is to inform others of the state of the system we test and how risky we think releasing it could be.
• Stick to the facts and keep it simple. By raising awareness of issues you could be seen as a troublemaker. You can limit backlash by framing your findings in terms of the risk to the company.
• Do not under any circumstances blame anyone.

Answer 2

First of all this is a managerial problem, not test per-se. Secondly there is no right answer here and it all depends on the context, maybe it's acceptable to release a lousy product, maybe it's safety critical product and you must stop it asap.

Always start by communicating information, so collect as much information as you can into a easy to understand dashboard/presentation/something and present it to your peers- in this case probably developers, developer managers and other related roles like product managers.

The results of the meeting could be anything from ignoring the problem (see next paragraph then), to having war room meetings like you suggested or simply delay the delivery deadline.

If you are not satisfied from the results of the meeting, and if you have the power to reject a build (this is a managerial decision, someone trusted you and your judgment) then go ahead and reject the build.

Answer 3

Start focusing on bug prevention, not on bug follow up and fire-fighting!

Get together with the team, select a bug as the subject and find answers to questions like: 'In what development phase should this bug have been found?', 'Why didn't we catch the bug at that time?', 'What should we do to prevent such a bug/so many bugs?' Possible answers are: 'Developers should write more unit/integration testing during development' or 'Let's start doing/do more Test Driven Development', 'The specs should be more explicit/less ambiguous', 'We need better logging facillities', 'We need to ask questions when things are not clear', etc. Repeat for a few more recurring or otherwise interesting bugs.

Call it an RCA (Root Cause Analysis), but many people associate that with a heavy-weight process. But if you focus on a few bugs and pose the relevant questions (like the ones I suggest above), you can turn it into a light-weight process where all parties involved may benefit from.

Answer 4

I agree with Kate's response. We need to triage on what is the final acceptable bug count, sort out the critical blockers and then get into the war room with all the stakeholders and decide on what is the acceptable count to be achieved for go/no-go decision.

Yes, Testing is not answerable but they have to ensure quality and their decision can impact a delivery big time.

Answer 5

For this before getting into the solution my mind battles for some questions like:

Why testers ain't doing smoke testing, before going straight to the testing? why developers not testing the code before delivering the code to testers?

Everyone knows 80% of the problems in software arise due to 20% of bugs, well one must prioritize tasks accordingly to control the scene of crime, so that problem somewhere get deficit in terms of no. of bugs.

Well, let that. The only solutions left behind to tackle such situation is to let the QA tester do their work with ample time to follow each and every aspect of functionality. Second, you can get assistance from the automation technology by hiring some expert automation testers. They will ease and fold the whole scenario.

Answer 6

There can be different scenarios which can be easily handeled inorder to avoid too many bugs at the later stage:

1. If the Management knows that the application is complex then Schedule should be relaxed so that QA team can get enough time for testing.

2. Use automated testing services for automating repetitive scenarios. This would ease up of functional engineers and enable them to focus on new scenarios.

3. If the QA team is failing even after the above two solutions then its time to get help of leading quality assurance services provider.