How do we deal with occasional (but real) application failures

Question

Background:

In our UAT testing environment, there have been a few intermittent bugs that come and go. I am not a permanent employee here, my current contract will finish early August.

What I have done

• Reported intermittent bugs on Jira with exact steps to reproduce them

Why am I concerned?

• Test Manager mentioned that re-test all intermittent bugs in five days, if they can not be reproduced, consider them resolved and close their respective Jira tickets.

• I am concerned as this re-test them in five days approach has no solid reason behind it. E.g. There is no indication our UAT testing environment will become more stable in five days; there is no major release to be deployed in UAT in five days. This five days is basically a random guess. There is no guarantee those intermittent bugs do not come back in the future.

Has anyone had experience in handling intermittent issues?

Answer 1

Intermittent problems are hard. They're hard to reproduce and hard to track down.

I work on a process with intermittent issues:

• Report the issue - I make sure to note that the issue is intermittent and that I will be researching it. I make a note to myself to watch out for the issue as well.
• Triage the issue - While I'm researching and noting what I'm doing when the issue happens, I look at how often it's likely to happen, whether it looks like a possible race condition, a data-bound issue, or a combination of the multiple interacting conditions a typical app needs to deal with (in my experience most intermittent issues fall into one of these categories - one really special one took well over 100 steps to reproduce and missing any of them made it not happen: but it was catastrophic when it did occur). I also look at the potential impact and whether any particular customers are more likely to encounter it because of their configuration.
• Log everything - If I can, I enable every form of logging the application and system has. If not, I try to reproduce while using a logging app that records my actions.
• Update the issue with findings - Each time I find more information, I will update the issue. My goal at this point is to gather as much data as I can so that when customers report the issue the development team is more likely to be able to fix it.

If I find a reliable way to reproduce the issue, I will add that to the report. If I find a process that will cause the issue within a set time period, I'll add that to the issue (I encountered one of these where after enough navigation through a large data set, the current position in the data set would be lost. Exactly how much navigation varied, but with a data set in the hundreds and a combination of moving forward and back through the pages and making edits, I could guarantee the problem would happen before reaching the last page).

I would disagree with your test manager. Five days is not enough time. I'd suggest that instead of marking them resolved, you mark them unable to reproduce. This keeps them searchable.

If the test manager insists, I'd suggest you include some key text people can easily search on in the issue when you mark it resolved. Something like [Intermittent - Unable to reproduce] in the title or another searchable field would work. Then tell your other team members so that if they run into an intermittent issue, they can search easily too.

Answer 2

Yes, often. Intermittent bugs are the most annoying to deal with, especially when you don't even know what triggered the problem.

Unless the underlying code/infrastructure changes within the five days (and often, even if it does), there's no technical reason to assume that the underlying defect doesn't exist any more, even if you can't recreate it.

However, there is a cost/benefit judgement that has to be made. If it's an intermittent problem that doesn't happen very often, and/or the impact is small, closing the bug as "unable to reproduce" may be appropriate. Personally, I'd prefer to leave it open, so that if the problem is seen again, people are more likely to connect whatever they saw/found with an already open problem, but there's no reason to keep it open if, for some cultural reason, open defects are seen as a problem.

I would consider it a problem if you're being asked to close the bug as "fixed," or something similar, because you have no reason to believe the problem was fixed, and it's going to make understanding the actual quality of the code base much more difficult. There's a big difference between something that has 100 fixed bugs, and one that has 50 fixed bugs and 50 "unable to reproduce" bugs.

Now, one thing that these bugs probably indicate is a need for better logging and/or error toleration/recovery code, but, again, there's a cost/benefit judgement that needs to be made.

In terms of handling them, you have to look at the impact, the frequency, and the expected usage. I spent a few weeks tracking down a security problem, for example, because it could have exposed data that shouldn't have been exposed, and our customers really care about that. Similarly, if you have something that only happens 1 of every 100,000 times you run a testcase, it may or may not be a problem, depending on how often you expect that function to be invoked.

[Edited to add] One way you could look at this, though you'd have to get buy-in, which might be difficult, is that if there's a problem, but whatever diagnostics you have don't give developers/whoever enough information to debug/fix the problem, that that is a problem in and of itself. We call it a First Failure Data Capture problem in my area, and it's at least sometimes treated as more of a problem than the problem that was unable to be debugged. Because a customer will be annoyed (or more), if there's a problem, but they tend to get really upset if there's a problem they have to keep recreating because you can't explain how it's happening. So if we open a defect, and it isn't able to be debugged, we'll open a second defect for the inability to debug the problem.

Answer 3

Against flaky test the Googlers themselves contend in vain.

So we mere mortals do what Michael Durrant and Kate Paulk suggest.

See also Heisenbug.

Answer 4

Intermittent Failures are the bane of our work.

It is important to know that they are:

• Common to many, if not most, organizations
• Most commonly UI tests that interact with a browser
• As common as 1 in 2 or as rare as 1 in 1 million

Avoiding them requires a number of different good practices:

• Ensuring that the database is as empty as possible and is reset after each test
• Wait techniques to make sure a browser page assets are all loaded
• Wait techniques to make sure a browser page is fully rendered
• Wait techniques to deal with ajax, i.e. implicit waits
• Allowing time for fixing tests now over pending the test and adding to the backlog
• Understanding that as test suites grow they start to require maintenance reources
• Understanding that different devices and browser versions have different issues to address
• Collect examples over time and then look for patterns - detective work that can take months*

* months or even years in some cases. So 5 days is not enough time by a long way. I would look to have a discussion with your manager where you say you need more examples than you'll get in 5 days in order to see the patterns.
Your manager has a point though, if these tests are sometimes failing it may make sense to actually remove them briefly and actually fix them. What happens in many organizations though is that they are removed and not fixed. The details count here. If a test is failing 1 in 3 runs, that is too often and is leading to lack of confidence in the test suite, plus that test might also miss other stuff it should still be checking. A failure rate of 1 in 10,000 however would probably lead me to leave the test in place and when I have no other work to do... ;) I'll fix it. It's always about company priorities, other work, time and expense, value and criticality of the test.

A fix in 5 days philosophy can be a good thing! However I would look to make sure that my workload for regular features and bugs is only about 40% of my time so that I have the time and resource to fix these issues. 20% of my time is maintaining tests so that means 40% is left to features and regular bugs. This comes from experience. It's really hard to accept / implement / communicate.