8

We had a software tester in our QA team that had been reporting a lot of issues, but most of them were low-priority visual bugs, browser-specific behavior bugs, sophisticated (but with low probability of happening) bugs involving using multiple concurrent users, poor network connection, insanely fast "cybersport-style" clicking etc.

On the other hand, he was constantly missing serious defects that were sometimes noticed too late on production resulting in "expensive" fixes.

So, to recap, he was doing his job - he was testing and reporting issues, but it looked like the problem was with his testing focus - instead of focusing on critical functionality of the application under test, he was concentrating on esoteric unusual use-cases.

How would you deal with this kind of situation?


In some sense, this is a different perspective on a How should a Software Tester deal with missed Defects/bugs in Production? question.

  • First and foremost, does this tester understand both the user perspective and what the purpose of the application is? Maybe they're focusing on these things because they don't really get what the happy path is? – ernie Jun 10 '17 at 23:31
  • @ernie I guess you are onto something here, I remember him struggling with getting the end purposes of the product and the company even though we've got over it multiple times..so, yeah, maybe he tried to focus on the things he was good at..sort of, follow the safe path - the path he knew he could deliver something at. Thanks, good question. – alecxe Jun 11 '17 at 0:34
12

I think their job should be to find significant issues that will affect the user and the business

I think that 'doing his job' means a lot more than clicking on every item and entering bad information in every field. It should be, within the context of the application, the industry, the user, the task, that there is a significant issue that affects the users ability to complete their task.

Some things I would consider doing:

  • Have a review session to go over how to categorize tickets, what is significant for your company. Do this mostly in the abstract before looking at any specific cases.
  • Review the categories you have tickets and make sure there are ways to label tickets as 'ui isssue', 'mouse clicking issue', etc.
  • Have a regularly scheduled 'bug review' session where you examine, in a lower stress environment bugs that were found and how they can be avoided in the future. Make this a team effort including devs, i.e. not pointing at QA folks.
  • Have a regularly scheduled 'bug review' session just within QA to review tickets before the above group review with the wider team.
  • Use approaches such as "3 why's" and root cause analysis to get to the true cause of issues.
  • Increase the focus (for everyone - analysts, devs, qa, etc) on testing and talking about testing and test plans earlier in the process - commonly referred to as 'shift left'. The best bugs are the ones that never happen because of a few key conversations about both functionality and testing up front before any code is written.
  • Make sure you have a definition of done and that it includes the broad areas that QA should be signing off on, e.g. happy flow, sad flow, different devices, accessibility testing, etc.
  • Seek ways to relate bugs to the company's mission. For example if a company measure daily revenue, consider analysis and AB testing to show monetary value of bugs being fixed. This involves a lot of analysis and setup but can be gold in terms of deciding what should be worked on.
  • Consider periodic backlog review meetings where you go through older tickets and see whether they are worth fixing or should just be closed. This helps to reinforce what the company consider worth logging.
  • Make sure the company metrics for quality are clearly defined and communicated.
| improve this answer | |
4

Categorize the bugs he has been missing and the bugs he has managed to find, what do they have in common?

  • For the bug he managed to find, low-priority visual bugs, browser-specific behavior bugs, sophisticated (but with low probability of happening) bugs involving using multiple concurrent users, poor network connection, insanely fast "cybersport-style" clicking etc.
  • The characteristics above may suggest he is a visual-orientated person and is capable of taking unusual ways to discover unusual bugs.

  • But it looked like the problem was with his testing focus. This tester may have a problem to prioritize tasks. He does not see critical functionality as critical.
  • What are other shared characteristics of the bug he keeps on missing?
  • Is there any training / mentoring that you can provide him in order help him overcome his shortcomings? This mentoring process may require an experienced tester to do pair testing with him and it may take some time.
| improve this answer | |
2

Did anyone review the testers test plan or test cases? During the review did anyone apply an 80/20 rule to the planned test cases?

Do you know the requirements the tester was planning to cover in their testing?

Do you have well defined requirements?

Does your tester actually know how to use the application? Have they ever asked you for help defining and designing test data?

Is your QA environment managed and maintained to ensure it matches your production environment? Were the issues from production reproducible in the QA environment.

Frankly, your question starts with a lot of finger pointing. I'm not saying your QA resource isn't at fault, but if you don't hold someone accountable for their work then you might not like the result. Did you state that you wanted a specific type of testing performed or a specific feature validated? If not, and your resource lacks experience/discipline, then you will likely get a lot of low priority or enhancement bugs.

| improve this answer | |
2

This is a question that should be asked whenever there is a production defect. Not for playing a blame game on a QA person. But to identify what went wrong in our software development and QA process.

In my previous job this is a point where we used to do root cause analysis and to do CAPA - corrective and preventive actions.

Also in the recent Agile and in Shift Left Test Strategy, I will strongly say that one cannot simply ask only the QA this question. This should be asked to a developer of that feature and also to the Lead developer who did the code review for this feature.

Also when doing the root cause analysis you cannot simply stop at the very first answer. You should continuously ask questions based on the answers.

For example:

How this defect got missed in QA? -We have not covered this scenario in our test case. -Also developer has not written proper Unit/Integration tests to cover this edge case.

Why this scenario has not been covered in the test case? and also Why this scenario has been missed out during test case review? Why developer has not written any unit/integration tests for this edge case?

-We have not got enough time to cover more scenarios or to do proper test case review or to write unit/integration test case.

Why we don't have much time to do this activity? In Sprint planning, we have taken lot of cases/feature from the backlog for this sprint which actually created some time constraints for both dev and QA.

We have taken features from the backlog based on our story points for each feature/story. So, Why we have not properly assigned story points for those features?

-Yes we should be more careful when assigning story points for the features/stories that we are gonna develop for every sprint.

So the final answer is the CAPA that we have to take a look at very carefully and it should be implemented for every sprint going forward.

Disclaimer: The answer is applicable only for people and organization who have got some broad mind to accept their own mistakes and willing to correct them as early as possible. This is not applicable for some companies/people who are more interested to play a political blame game or where they want to treat QA as Scapegoats for the production defects.

| improve this answer | |
New contributor
Govinda Hinduja is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
1

All Bugs are Good

If we re-define the bug queue to be that of a database that logs behaviors given specific input and then adopt the concept that "more data in the database is better", we have a better way to assess "most value to the product" within the current and future iterations.

In other words, foster a "Write as many bugs as you can find because this is ultimately good". All data in the queue is reviewed and prioritized continually. All bugs being rated with focus only on those most important to the product. Those that are not rated high enough are moved to the backlog, out of view, but not forgotten.

Depth of Tests must Continuously Improve

At some point, the smaller more trivial bugs would no longer be logged once they are in the database because "We don't write duplicate Bugs". This means that the depth of tests must improve. Therein lay the solution: All tests should be reviewed and approved continuously too. Making each Test Plan (collection of tests) approved at iteration planning sessions.

The Test Plan focuses only on those User stories and Bugs that are most value to the product for any given iteration. By default, each iteration's Test Plan itself is continuously improving simply because the process demands it.

Overall methodology must Improve

This, in turn, improves the overall Processes and Policies the QA Team establishes as they go, making the best way to operate standardized. If gaps are found (as you mentioned) then perhaps a bug is written against the process itself, that bug could be something as simple as "Create additional training or teach new methods within the given context".

Continuous Improvement of all aspects of Test has profound implications.

| improve this answer | |
0

Missing defect that are spotted on Live environment can be detrimental to a software qa company. And in case, a software tester is missing these regularly then below steps could be taken to isolate the cause and improve.

  • Tester should have requirement document for the functionality to be implemented.
  • Tester should have complete understanding of the application in use.
  • Detailed test and use cases should be made with priority mentioned for the application.
  • Tester should know how to perform integration testing if multiple modules are being used.
  • Tester should regress the fixed defects along with testing the related areas of the fixed functionality.
  • Sanity testing should be performed before and after the application is deployed on live environment.
  • A staging environment should be used to check if functionality is implemented and tested correctly, and the same is being reflected once go ahead is sent on QA environment.
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.