What to do if a Software Tester is constantly missing Defects/bugs in Production?

Question

We had a software tester in our QA team that had been reporting a lot of issues, but most of them were low-priority visual bugs, browser-specific behavior bugs, sophisticated (but with low probability of happening) bugs involving using multiple concurrent users, poor network connection, insanely fast "cybersport-style" clicking etc.

On the other hand, he was constantly missing serious defects that were sometimes noticed too late on production resulting in "expensive" fixes.

So, to recap, he was doing his job - he was testing and reporting issues, but it looked like the problem was with his testing focus - instead of focusing on critical functionality of the application under test, he was concentrating on esoteric unusual use-cases.

How would you deal with this kind of situation?

---

In some sense, this is a different perspective on a How should a Software Tester deal with missed Defects/bugs in Production? question.

Answer 1

I think their job should be to find significant issues that will affect the user and the business

I think that 'doing his job' means a lot more than clicking on every item and entering bad information in every field. It should be, within the context of the application, the industry, the user, the task, that there is a significant issue that affects the users ability to complete their task.

Some things I would consider doing:

• Have a review session to go over how to categorize tickets, what is significant for your company. Do this mostly in the abstract before looking at any specific cases.
• Review the categories you have tickets and make sure there are ways to label tickets as 'ui isssue', 'mouse clicking issue', etc.
• Have a regularly scheduled 'bug review' session where you examine, in a lower stress environment bugs that were found and how they can be avoided in the future. Make this a team effort including devs, i.e. not pointing at QA folks.
• Have a regularly scheduled 'bug review' session just within QA to review tickets before the above group review with the wider team.
• Use approaches such as "3 why's" and root cause analysis to get to the true cause of issues.
• Increase the focus (for everyone - analysts, devs, qa, etc) on testing and talking about testing and test plans earlier in the process - commonly referred to as 'shift left'. The best bugs are the ones that never happen because of a few key conversations about both functionality and testing up front before any code is written.
• Make sure you have a definition of done and that it includes the broad areas that QA should be signing off on, e.g. happy flow, sad flow, different devices, accessibility testing, etc.
• Seek ways to relate bugs to the company's mission. For example if a company measure daily revenue, consider analysis and AB testing to show monetary value of bugs being fixed. This involves a lot of analysis and setup but can be gold in terms of deciding what should be worked on.
• Consider periodic backlog review meetings where you go through older tickets and see whether they are worth fixing or should just be closed. This helps to reinforce what the company consider worth logging.
• Make sure the company metrics for quality are clearly defined and communicated.

Answer 2

Categorize the bugs he has been missing and the bugs he has managed to find, what do they have in common?

• For the bug he managed to find, low-priority visual bugs, browser-specific behavior bugs, sophisticated (but with low probability of happening) bugs involving using multiple concurrent users, poor network connection, insanely fast "cybersport-style" clicking etc.
• The characteristics above may suggest he is a visual-orientated person and is capable of taking unusual ways to discover unusual bugs.

---

• But it looked like the problem was with his testing focus. This tester may have a problem to prioritize tasks. He does not see critical functionality as critical.
• What are other shared characteristics of the bug he keeps on missing?
• Is there any training / mentoring that you can provide him in order help him overcome his shortcomings? This mentoring process may require an experienced tester to do pair testing with him and it may take some time.

Answer 3

Did anyone review the testers test plan or test cases? During the review did anyone apply an 80/20 rule to the planned test cases?

Do you know the requirements the tester was planning to cover in their testing?

Do you have well defined requirements?

Does your tester actually know how to use the application? Have they ever asked you for help defining and designing test data?

Is your QA environment managed and maintained to ensure it matches your production environment? Were the issues from production reproducible in the QA environment.

Frankly, your question starts with a lot of finger pointing. I'm not saying your QA resource isn't at fault, but if you don't hold someone accountable for their work then you might not like the result. Did you state that you wanted a specific type of testing performed or a specific feature validated? If not, and your resource lacks experience/discipline, then you will likely get a lot of low priority or enhancement bugs.

Answer 4

This is a question that should be asked whenever there is a production defect. Not for playing a blame game on a QA person. But to identify what went wrong in our software development and QA process.

In my previous job this is a point where we used to do root cause analysis and to do CAPA - corrective and preventive actions.

Also in the recent Agile and in Shift Left Test Strategy, I will strongly say that one cannot simply ask only the QA this question. This should be asked to a developer of that feature and also to the Lead developer who did the code review for this feature.

Also when doing the root cause analysis you cannot simply stop at the very first answer. You should continuously ask questions based on the answers.

For example:

How this defect got missed in QA? -We have not covered this scenario in our test case. -Also developer has not written proper Unit/Integration tests to cover this edge case.

Why this scenario has not been covered in the test case? and also Why this scenario has been missed out during test case review? Why developer has not written any unit/integration tests for this edge case?

-We have not got enough time to cover more scenarios or to do proper test case review or to write unit/integration test case.

Why we don't have much time to do this activity? In Sprint planning, we have taken lot of cases/feature from the backlog for this sprint which actually created some time constraints for both dev and QA.

We have taken features from the backlog based on our story points for each feature/story. So, Why we have not properly assigned story points for those features?

-Yes we should be more careful when assigning story points for the features/stories that we are gonna develop for every sprint.

So the final answer is the CAPA that we have to take a look at very carefully and it should be implemented for every sprint going forward.

Disclaimer: The answer is applicable only for people and organization who have got some broad mind to accept their own mistakes and willing to correct them as early as possible. This is not applicable for some companies/people who are more interested to play a political blame game or where they want to treat QA as Scapegoats for the production defects.

Answer 5

All Bugs are Good

If we re-define the bug queue to be that of a database that logs behaviors given specific input and then adopt the concept that "more data in the database is better", we have a better way to assess "most value to the product" within the current and future iterations.

In other words, foster a "Write as many bugs as you can find because this is ultimately good". All data in the queue is reviewed and prioritized continually. All bugs being rated with focus only on those most important to the product. Those that are not rated high enough are moved to the backlog, out of view, but not forgotten.

Depth of Tests must Continuously Improve

At some point, the smaller more trivial bugs would no longer be logged once they are in the database because "We don't write duplicate Bugs". This means that the depth of tests must improve. Therein lay the solution: All tests should be reviewed and approved continuously too. Making each Test Plan (collection of tests) approved at iteration planning sessions.

The Test Plan focuses only on those User stories and Bugs that are most value to the product for any given iteration. By default, each iteration's Test Plan itself is continuously improving simply because the process demands it.

Overall methodology must Improve

This, in turn, improves the overall Processes and Policies the QA Team establishes as they go, making the best way to operate standardized. If gaps are found (as you mentioned) then perhaps a bug is written against the process itself, that bug could be something as simple as "Create additional training or teach new methods within the given context".

Continuous Improvement of all aspects of Test has profound implications.

Answer 6

Missing defect that are spotted on Live environment can be detrimental to a software qa company. And in case, a software tester is missing these regularly then below steps could be taken to isolate the cause and improve.

• Tester should have requirement document for the functionality to be implemented.
• Tester should have complete understanding of the application in use.
• Detailed test and use cases should be made with priority mentioned for the application.
• Tester should know how to perform integration testing if multiple modules are being used.
• Tester should regress the fixed defects along with testing the related areas of the fixed functionality.
• Sanity testing should be performed before and after the application is deployed on live environment.
• A staging environment should be used to check if functionality is implemented and tested correctly, and the same is being reflected once go ahead is sent on QA environment.

Answer 7

Try to find out the hidden factor behind the bad testing, and structurally fix it so that this person gets on the right track and that other team members don't follow the same mistake.

1. Go back to your defect repository and find out which defects were missing
2. Then link them to your test case repository and find out what kind of tests were missing in the test cases.
3. Try to find out 2-3 repeating patterns: is there a module that is being missed more frequently that others, or is he missing on some kind of requirements/ functionalities, or any other trend.
4. Have a team session and tell your team about the weak area and that you want to strengthen that with a more thorough test coverage
5. Revise the test case repository and review/ approve the test case yourself in next testing cycles to make sure that you are not missing those type of test cases

To be able do the above analysis and action, you should have a easily accessible and maintainable test case repository, a system by which you can approve the test cases for a new test cycle, and a system of re-usability of test cases in next test cycle within your project.

There a bunch of test management tools out there that can help you. TestLink is open source, is nice in test case re-usability but doesn't let you link your test cycles with defects. HP ALM is a comprehensive enterprise grade tool with all the functions you can think of but has a high learning curve and may seem costly if you are not a very large organization. Kualitee is a relatively lesser known and an all rounder type of tool that will let you maintain test cases repository, let you approve new test cases and is relatively simple to use.