5

We had a software tester in our QA team that had been reporting a lot of issues, but most of them were low-priority visual bugs, browser-specific behavior bugs, sophisticated (but with low probability of happening) bugs involving using multiple concurrent users, poor network connection, insanely fast "cybersport-style" clicking etc.

On the other hand, he was constantly missing serious defects that were sometimes noticed too late on production resulting in "expensive" fixes.

So, to recap, he was doing his job - he was testing and reporting issues, but it looked like the problem was with his testing focus - instead of focusing on critical functionality of the application under test, he was concentrating on esoteric unusual use-cases.

How would you deal with this kind of situation?


In some sense, this is a different perspective on a How should a Software Tester deal with missed Defects/bugs in Production? question.

  • First and foremost, does this tester understand both the user perspective and what the purpose of the application is? Maybe they're focusing on these things because they don't really get what the happy path is? – ernie Jun 10 '17 at 23:31
  • @ernie I guess you are onto something here, I remember him struggling with getting the end purposes of the product and the company even though we've got over it multiple times..so, yeah, maybe he tried to focus on the things he was good at..sort of, follow the safe path - the path he knew he could deliver something at. Thanks, good question. – alecxe Jun 11 '17 at 0:34
10

I think their job should be to find significant issues that will affect the user and the business

I think that 'doing his job' means a lot more than clicking on every item and entering bad information in every field. It should be, within the context of the application, the industry, the user, the task, that there is a significant issue that affects the users ability to complete their task.

Some things I would consider doing:

  • Have a review session to go over how to categorize tickets, what is significant for your company. Do this mostly in the abstract before looking at any specific cases.
  • Review the categories you have tickets and make sure there are ways to label tickets as 'ui isssue', 'mouse clicking issue', etc.
  • Have a regularly scheduled 'bug review' session where you examine, in a lower stress environment bugs that were found and how they can be avoided in the future. Make this a team effort including devs, i.e. not pointing at QA folks.
  • Have a regularly scheduled 'bug review' session just within QA to review tickets before the above group review with the wider team.
  • Use approaches such as "3 why's" and root cause analysis to get to the true cause of issues.
  • Increase the focus (for everyone - analysts, devs, qa, etc) on testing and talking about testing and test plans earlier in the process - commonly referred to as 'shift left'. The best bugs are the ones that never happen because of a few key conversations about both functionality and testing up front before any code is written.
  • Make sure you have a definition of done and that it includes the broad areas that QA should be signing off on, e.g. happy flow, sad flow, different devices, accessibility testing, etc.
  • Seek ways to relate bugs to the company's mission. For example if a company measure daily revenue, consider analysis and AB testing to show monetary value of bugs being fixed. This involves a lot of analysis and setup but can be gold in terms of deciding what should be worked on.
  • Consider periodic backlog review meetings where you go through older tickets and see whether they are worth fixing or should just be closed. This helps to reinforce what the company consider worth logging.
  • Make sure the company metrics for quality are clearly defined and communicated.
3

Categorize the bugs he has been missing and the bugs he has managed to find, what do they have in common?

  • For the bug he managed to find, low-priority visual bugs, browser-specific behavior bugs, sophisticated (but with low probability of happening) bugs involving using multiple concurrent users, poor network connection, insanely fast "cybersport-style" clicking etc.
  • The characteristics above may suggest he is a visual-orientated person and is capable of taking unusual ways to discover unusual bugs.

  • But it looked like the problem was with his testing focus. This tester may have a problem to prioritize tasks. He does not see critical functionality as critical.
  • What are other shared characteristics of the bug he keeps on missing?
  • Is there any training / mentoring that you can provide him in order help him overcome his shortcomings? This mentoring process may require an experienced tester to do pair testing with him and it may take some time.
2

Did anyone review the testers test plan or test cases? During the review did anyone apply an 80/20 rule to the planned test cases?

Do you know the requirements the tester was planning to cover in their testing?

Do you have well defined requirements?

Does your tester actually know how to use the application? Have they ever asked you for help defining and designing test data?

Is your QA environment managed and maintained to ensure it matches your production environment? Were the issues from production reproducible in the QA environment.

Frankly, your question starts with a lot of finger pointing. I'm not saying your QA resource isn't at fault, but if you don't hold someone accountable for their work then you might not like the result. Did you state that you wanted a specific type of testing performed or a specific feature validated? If not, and your resource lacks experience/discipline, then you will likely get a lot of low priority or enhancement bugs.

1

All Bugs are Good

If we re-define the bug queue to be that of a database that logs behaviors given specific input and then adopt the concept that "more data in the database is better", we have a better way to assess "most value to the product" within the current and future iterations.

In other words, foster a "Write as many bugs as you can find because this is ultimately good". All data in the queue is reviewed and prioritized continually. All bugs being rated with focus only on those most important to the product. Those that are not rated high enough are moved to the backlog, out of view, but not forgotten.

Depth of Tests must Continuously Improve

At some point, the smaller more trivial bugs would no longer be logged once they are in the database because "We don't write duplicate Bugs". This means that the depth of tests must improve. Therein lay the solution: All tests should be reviewed and approved continuously too. Making each Test Plan (collection of tests) approved at iteration planning sessions.

The Test Plan focuses only on those User stories and Bugs that are most value to the product for any given iteration. By default, each iteration's Test Plan itself is continuously improving simply because the process demands it.

Overall methodology must Improve

This, in turn, improves the overall Processes and Policies the QA Team establishes as they go, making the best way to operate standardized. If gaps are found (as you mentioned) then perhaps a bug is written against the process itself, that bug could be something as simple as "Create additional training or teach new methods within the given context".

Continuous Improvement of all aspects of Test has profound implications.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.