How do you manage the expectations of the management that the QA team would find all the bugs in their owned feature?

Question

In a QA team suppose someone finds a defect that doesn't belong to his own assigned feature area but actually belongs to my owned feature area. That person logs this defect in the tracking system. It is not really a basic defect (sanity tests wouldn't catch it) but it isn't very corner case either.

Now the management sees it as a bad thing. They contact me and wonder why the defect was missed by me. After all I am owning this feature area so they expect that I should have found that defect and not anyone else. I don't feel this expectation is realistic. Now I need to explain to the management that I missed that defect because I was busy on a different QA activity.

As QA members we are involved in, at the very least, any 2 of the following activities daily:

• running sanity tests almost daily (manual) (it could be on any of the pods for that day - qa pods, staging pods, production pods)
• testing hotfixes that make their way into the production pod
• working on customer issues that arise
• performing full test case regression on your owned area (takes 2 weeks)
• going through ERs/new features that are upcoming in the next version and understanding them
• writing test cases for the ERs/getting them reviewed
• executing the ER test cases
• exploring defects when discoverd and logging issues with details
• defect verification and further sanity tests for them as the fixes get merged into the code

So here are my questions:

1. Firstly, are the management's expectation realistic here?
2. If not, then how do I explain it to them? If I tell them whatever I have written here it seems to me that they are taking it as an employee being inefficient.

Answer 1

Let's put aside all defensiveness for a moment. Why in fact was the defect not found? Sometimes, "because I have too many other things to do" is in reality "because the self-set priority of tasks wasn't appropriate" or "because my testing isn't deep enough to find that problem", or something else within your control.

Be careful when arguing that you have too much to do - that's a weak argument that you will seldom win. Everyone has too much to do, and not enough time. And every manager expects you to understand what is most important, and what is "nice to have".

Instead, think about what would have allowed you to find that bug before anyone else. Perhaps far stronger sanity tests, or earlier regression tests, or something else that is within your control?

Try to approach your discussion with management with a non-defensive attitude, and perhaps with a proposed solution. Your management will appreciate that.

One thing you might want to consider is reading "Perfect Software: And Other Illusions About Testing" by Gerald Weinberg beforehand. Some good discussion points in there. Just remember, management isn't asking you why the software isn't perfect. Instead they are asking you why you didn't find a bug in your particular part of the software that they (rightly or wrongly) feel you should have caught. Perhaps they are justified in their feeling, perhaps not. Try to dig in and find out for yourself if they are right.

Answer 2

Given infinite time and infinite resources you could never find all the defects within a feature or product, there may be some combination of keystrokes or conditions that can cause issues but are not always reproducible. If Management is expecting you to find any and all defects in a feature then they are giving you unrealistic goals, if they allow that you can adapt your process to catch the defects by adjusting your testing as items "get by" then you can keep improving and be sure that "this one" won't get by again.

What you can explain to your Management, and using any high profile outage or defect that happens, is that testing is an adaptive or evolutionary process. Even Microsoft releases patches against defects that are continually found in their software, if Microsoft is not finding every defect how are you expected to? You can't find everything, especially if you are performing multiple test types and methods, but you can always reassess your testing and improve what you find. In this case if the defect got out and was found by someone else ask the important questions, why was this missed? Why did it take someone not familiar with the product to find it? What test case can you add to assure it won't happen again?

What it will ultimately come down to is how willing is your Management to listen to you and your concerns and allow that your methodology will work, but will always need slight adjustments to keep things in check?

Answer 3

In addition to what everyone else has said, here are some other factors to consider:

• Is your manager aware of the complexity level of your software? Given that your team is assigned to areas of the product, it sounds like you have a highly complex piece of software to support, which means that it's physically impossible to catch everything. If management isn't aware of the impact of the complexity level, you may need to provide a few examples for them to see what's actually happening.
• Is your management aware of what you and your fellow testers do on a daily basis? If not, this would be a good opportunity to educate them: if they see your team as a black hole where code goes in and quality (hopefully) comes out, they're likely to think you work miracles on a daily basis (you probably do, but not the kind your management thinks you're doing).
• Do you have a "Speaker to Management"? In my experience, people in managerial roles don't speak the same language as testers do. Both of them sound like English, but the meanings of a lot of the words are different.
• Can you explain the power of serendipity and exploratory testing to your management or your Speaker to Management? From your description, it sounds like the problem that was found was something your regular testing schedule would not have found or would not have found until much later in the cycle: that means that someone else was doing something, noticed something odd and thought "Hmm. That's funny. I wonder what happens if...?" and found a bug. This is - as you recognized - a good thing, but because it's not a planned activity it's the kind of thing someone with a management background might not really understand without some leadup and background.
• Can you point to both a full schedule and a test case that would have found this problem at some point in the cycle? While this is anatomy-covering on your part, you can also use the information to make the case that your team is understaffed so any problems anyone finds are a bonus to you. (I don't know if this is the case, but it's something I'm dealing with at the moment, so I'm familiar with that particular argument).

I hope some of these suggestions help.

Answer 4

It seems to imply management in your org expects the QE team to be the gatekeeper as it should not become a blame game of why the defect was missed rather why it was introduced in the first place and the team collectively investigate the root cause of it surfacing. This would be followed by how we can prevent such bugs from seeping through in the future.

Answer 5

We strive to find every defect, but for myriad reasons, bugs always slip through. Generally speaking, it is unrealistic to (1) expect a feature to be defect-free by the time customers start using it or (2) expect you to be the only person who finds defects in your feature.

Taken in isolation, your manager's reaction sounds unreasonable, but there may be other factors to consider. For example,

• Has this happened to you with this manager before?
• Has it happened to others in your QA team?
• Did your manager have a bad experience with a tester who was inefficient?
• Have you given your manager any reason to believe that you are inefficient?
• Is your manager under pressure to improve his team's quality or productivity?
• Is your company under pressure from your customers to improve quality?
• How complicated/testable is your feature compared to other features?

There may also be cultural issues at work that are unique to a work environment in India. In any case, understanding more of the underlying causes of your manager's reaction may help you formulate an appropriate response.

Answer 6

Ask them why the developer created the defect and didn't tell you!

Your managements expectations are unrealistic, but they are not uncommon. We deal with this on a weekly basis. You can never find every bug because you can never complete every test. Just a little something I learned from Kem Caner and James Bach.