20

The QA team starts testing a software/product and there are “way too many” defects…!
Every other scenario is failing, new flows are explored & clarifications sought.
What would be the strategy now?
It’s a complex application with tight schedule. Too many defects add cherry on the cake. The blame game starts. But at the end, development & QA are expected to work as a team & deliver the product/software.
What are the options?

• Reject the build. But that would mean delay in delivery.
• Get into a war room, daily. It helps to triage the defects & increase the seriousness.

What are the positive approaches at this condition?

  • 20
    Release or delay is a business decision. Why do you think it's QA responsibility to make the call? QA is not the only partner in the team responsible for quality. – Peter M. - stands for Monica Feb 6 '18 at 14:23
  • 1
    Who's strategy are you looking for? This is not a 'QA' problem. This looks like a team problem. And a company problem. The powers that be have to decide. That is why they earn the big paychecks (in most cases)... – Ray Oei Feb 6 '18 at 17:00
  • 1
    Related: sqa.stackexchange.com/questions/28126/… – Milo P Feb 6 '18 at 18:40
  • 1
    It's important to have an in-house person with detailed stakeholder knowledge AND internal authority to make informed triaging decisions. Battling everything out between QA and dev team creates incredible toxicity that poisons the workplace for years. This person of authority can be the QA head itself, but he/she must a) not already have poisoned the well regarding the dev team due to past failures b) be able to take the stakeholder point of view (which is a problem with very technical/bureaucratic QA heads). – pmf Feb 7 '18 at 16:23
29

It depends

I've been in this situation more times than I care to remember, and the general process I use works this way:

  1. Triage all findings - At this stage I would be working to triage all bug findings into 4 classes:
    • Blockers - any bug that makes it impossible for the product to perform the basic user acceptance tests, or breaks crucial functionality. The kind of bugs that fall into this category are things like being unable to use the feature that's being developed, the application crashes each time you try to access the new feature, you can't actually purchase anything in a webstore... the kind of bug the completely ruins the software.
    • Critical - any other bug that has a high risk of occurring live and a high impact. Inaccurate financial calculations fall here, as do major security flaws. They don't stop users, but they have serious negative impacts.
    • Moderate - Bugs that could occur and would cause embarrassing and/or expensive consequences if they happen, but which don't break any laws or cause serious, blocking problems.
    • Minor - Anything that doesn't fall into the other categories - these are the bugs that can wait.
  2. Report Triaged Findings - this is where things can get challenging. Once you and your team agree on the status of the bugs you are reporting, you need to provide your superiors with a short summary of your findings. The format I prefer is something along these lines:
    • Number of Blockers - The number of blocking issues currently not fixed. I'll often give this number in bold red text.
    • Number of Critical Bugs - again, this is the count of bugs that haven't been fixed.
    • State of Release - for this I generally use a three color system. Red means in my view there is no way the targeted deadline can be met without releasing a blocker. Yellow means that the deadline is possible but at risk and there may be critical bugs released. Green means that everything is on target to release without any blockers or critical bugs. I'll typically start sending this once a week as soon as I'm aware a targeted release is at risk. In the last week to 2 weeks before release (depending on how bad things are), I'll send the message daily.
  3. Escalate Findings - the first recipient of my triaged findings is generally my manager. If I don't see any attempt to fix the blockers, I'll start including my manager's manager. The timeline for escalation depends on the timeline to release and how bad things are. I have escalated all the way to the company CEO, which ultimately forced that particular planned release to become a beta (there were contractual requirements).

Some things that are worth remembering:

  • We don't have go/no-go authority. There may be contracted requirements to release on a particular date.
  • We are not gatekeepers. Our role is to inform others of the state of the system we test and how risky we think releasing it could be.
  • Stick to the facts and keep it simple. By raising awareness of issues you could be seen as a troublemaker. You can limit backlash by framing your findings in terms of the risk to the company.
  • Do not under any circumstances blame anyone.
  • Are critical bugs not blockers? Or are blockers not critical? It seems like a distinction without a difference. – Mehrdad Feb 7 '18 at 7:38
  • 2
    @Mehrdad I guess it means that if there's not enough time, you can still deliver a product while having critical issues, but certainly not blockers. Many found security flaws on browsers are criticals, but those don't stop users to browse websites. – Andrew T. Feb 7 '18 at 8:35
  • 1
    @AndrewT.: Maybe her characterization of "critical" is wrong then, because it sure seems to me that financial miscalculations and illegal behavior (her examples) would be blockers... – Mehrdad Feb 7 '18 at 8:46
  • 1
    The short-short version is you can deliver a working beta with critical bugs but not blockers. With blockers you can't do that much. – Kate Paulk Feb 7 '18 at 12:26
7

First of all this is a managerial problem, not test per-se. Secondly there is no right answer here and it all depends on the context, maybe it's acceptable to release a lousy product, maybe it's safety critical product and you must stop it asap.

Always start by communicating information, so collect as much information as you can into a easy to understand dashboard/presentation/something and present it to your peers- in this case probably developers, developer managers and other related roles like product managers.

The results of the meeting could be anything from ignoring the problem (see next paragraph then), to having war room meetings like you suggested or simply delay the delivery deadline.

If you are not satisfied from the results of the meeting, and if you have the power to reject a build (this is a managerial decision, someone trusted you and your judgment) then go ahead and reject the build.

  • 2
    To add to this, I would not just say "communicate information", but also make sure that it's clearly recorded. The impact of every bug must be clearly stated, so that if management go ahead with the release then they're explicitly accepting the consequences. And the decision to release must also be a formal management decision. Ultimately there must be one person or one group of people who owns that decision. As the QA person, you are not part of that group - you simply supply the information. – Graham Feb 7 '18 at 13:09
2

Start focusing on bug prevention, not on bug follow up and fire-fighting!

Get together with the team, select a bug as the subject and find answers to questions like: 'In what development phase should this bug have been found?', 'Why didn't we catch the bug at that time?', 'What should we do to prevent such a bug/so many bugs?' Possible answers are: 'Developers should write more unit/integration testing during development' or 'Let's start doing/do more Test Driven Development', 'The specs should be more explicit/less ambiguous', 'We need better logging facillities', 'We need to ask questions when things are not clear', etc. Repeat for a few more recurring or otherwise interesting bugs.

Call it an RCA (Root Cause Analysis), but many people associate that with a heavy-weight process. But if you focus on a few bugs and pose the relevant questions (like the ones I suggest above), you can turn it into a light-weight process where all parties involved may benefit from.

1

I agree with Kate's response. We need to triage on what is the final acceptable bug count, sort out the critical blockers and then get into the war room with all the stakeholders and decide on what is the acceptable count to be achieved for go/no-go decision.

Yes, Testing is not answerable but they have to ensure quality and their decision can impact a delivery big time.

1

For this before getting into the solution my mind battles for some questions like:

Why testers ain't doing smoke testing, before going straight to the testing? why developers not testing the code before delivering the code to testers?

Everyone knows 80% of the problems in software arise due to 20% of bugs, well one must prioritize tasks accordingly to control the scene of crime, so that problem somewhere get deficit in terms of no. of bugs.

Well, let that. The only solutions left behind to tackle such situation is to let the QA tester do their work with ample time to follow each and every aspect of functionality. Second, you can get assistance from the automation technology by hiring some expert automation testers. They will ease and fold the whole scenario.

  • Please do not link to your site without explicitly stating your affiliation to the site. This is the second time you have linked to something from your organization without mentioning your affiliation. – Kate Paulk May 18 '18 at 15:01
  • Please tell me how to do this – Munish Garg Sep 28 '18 at 9:45
  • All you have to do is say "Disclaimer: I am employed by [insert organization name]" – Kate Paulk Sep 28 '18 at 11:26
0

There can be different scenarios which can be easily handeled inorder to avoid too many bugs at the later stage:

  1. If the Management knows that the application is complex then Schedule should be relaxed so that QA team can get enough time for testing.

  2. Use automated testing services for automating repetitive scenarios. This would ease up of functional engineers and enable them to focus on new scenarios.

  3. If the QA team is failing even after the above two solutions then its time to get help of leading quality assurance services provider.

Not the answer you're looking for? Browse other questions tagged or ask your own question.