What is a good test review process for sporadic failures

Question

A conversation has been started recently in my company, about the report of sporadic test failures.

Every QA team has different processes. In our team for example, the test review occurs in weekly shifts. In additional, every QA group has different frequency of the test runs. Some jobs(group of tests) could run daily or two/three times per group.

The QA engineer has the responsibility to review, and open a bug ticket or a ticket for the internal tracking system. However due to various reasons there are cases where a test fails, and a regular restart make it pass (which for some it just fine as there is not a product bug). And many of those cases are occurring sporadically.

My approach usually is to file a ticket for tracking and go on with the next test. Even if i know that it is insignificant I would rather track it down. When I finish with the review I can go back and investigate or fix or whatever.

But there is another side which insists that Creating tickets for ANY single failure could not be efficient, as some of them could be really sporadic and might not be even reproducible. The reasoning I got was because if you create a ticket for something that is not reproducible, most likely that ticket will be hanging there forever and no one will care about that ticket. But I think that this is a team's process flaw and not about the review.

I would like to know what do you think about the handling of sporadic failures and advice for good QA process in such cases. How do you deal with sporadic failures and test stability?

Answer 1

From what you have written, I'm not certain if you mean bugs in the test code, or bugs in the product. I think it's important to distinguish between the two.

Every QA team has different processes.

Not a problem to me. If something works in one team, why force them to change?

due to various reasons there are cases where a test fails, and a regular restart make it pass (which for some it just fine as there is not a product bug)

If it's a problem with a check, go fix it. If it's a problem with the product, share the observation with the team (e.g. by adding a new bug in a bug tracking system, or by bringing it up on a daily stand-up).

However, be wary of such intermittent problems that go away with the next execution. They might point to something you are not aware of now, e.g. problems with web performance - maybe the application causes many long tasks in the browser, thus the browser has no resources to keep parsing, rendering, and painting. Or too much data goes over the network, so there is a delay because the browser is waiting for some resources. Such problems might cause such intermittent problems in functional automated checks.

My approach usually is to file a ticket for tracking and go on with the next test. Even if i know that it is insignificant i would rather track it down. When I finish with the review I can go back and investigate or fix or whatever.

Investigation of problems should happen before you raise a new bug. It's part of your job as a tester to try to be more specific as to what's wrong with the product. That investigation might reveal it's not a bug in the product, but rather with the automated check, then you should do something about it (fix it, not run the check, ...) - either way, you will be the one working on the fix, therefore, it might not be necessary to add this bug in any bug tracking system.

Creating tickets for ANY single failure could not be efficient...

Yes, it might not be. Sometimes it's enough to tell the developer who sits next to you, he can fix it immediately. Or you can share your observations in a meeting, in a stand up meeting, on a Slack call.

... as some of them could be really sporadic and might not be even reproducible

But not for these reasons! Intermittent or sporadic bugs can still have severe consequences, so you should investigate. It's a good idea to share your observations as well, tickets are one way to do it.

most likely that ticket will be hanging there forever and no one will care about that ticket

That's a speculation. Don't rely on such assumptions as a tester.

I would like to know what do you think about the handling of sporadic failures and advice for good QA process in such cases.

I investigate every bug. If it's a bug in the product, I share my feedback. If it's a bug in my testing code, I fix it and many times don't share this information with anyone.

How do you deal with sporadic failures and test stability?

These are two questions to me. Sporadic failures - I investigate, I share my observations with the team. Test stability - I care about my test code, I try to improve test stability, it's my personal responsibility as a coder of automated checks.

Answer 2

Only a consistently manually reproducible bug with specific given steps should be logged in the system.

If a bug found through manual or automated testing is not consistently reproducible then it should be investigated more by QA engineer to come up with definite minimal reproducible example.

Specifically with automation, there could be good number of flaky test cases which sometimes fail/pass seemingly without any explicit reason however if a QA engineer carefully reason and observe the application behavior during execution with the aid of simple debugging tools like printing on console can discover the reason which are mostly due to sync issues between application & test automation.

For a good healthy process to remove flakyness from tests I would suggest:

• Have small tests with few test steps
• Take lot of screenshots at all significant steps in the test
• Have Independent tests without any data dependencies between tests.
• Assert application objects state at each important step in test like no errors of any kind on save/submit on pages.