How can I isolate bugs of users that we are not able to replicate?

Question

I am having a recurring problem where bugs are cropping up once or twice in a build, and are reported by users but I and my team are not able to replicate them.

What techniques do you use to replicate bugs that are hard to replicate?

We are using Sentry which picks up exceptions but not glitches, which are the most frustrating bugs to solve since they appear once and reappear infrequently.

Answer 1

You have

intermittent failures

Welcome to the crowd. In my experience they are the norm and the bane of Quality Engineering.
So first off - accept that and start making resources available for it.

Some techniques that you can use:

• Allow sufficient resources. Be prepared to need to take days/weeks to fix intermittent failures
• Collect examples. Some intermittent failures don't reveal patterns until I see dozens of them
• Run a test locally multiple times. At least 100, if not 1000 times. Look for patterns
• Review the test setups. Test should not be sharing any data, even reference data
• For tests, use implicit wait for elements not explicit waits (sleeps)
• Use and enhance logs to show more info for specific incidents
• Run a test 100 or 1000 times on your ci server

Useful information for investigations that I have collected over time:

• Browser
• User type
• Use of ajax
• Time of day
• Screen size
• Browser version
• Use of javascript
• Frequency over time
• Various resource ids
• Function being performed
• Relation to release pushes / reboots

Answer 2

"How to Investigate Intermittent Problems" by James Bach provides a very thorough treatment of the subject. Here's an outline of his 92 suggestions:

If the bug exists, it has a cause:

Possibility 1: The system is NOT behaving differently. The apparent intermittence is an artifact of the observation.

Possibility 2: The system behaved differently because it was a different system.

Possibility 3: The system behaved differently because it was in a different state.

Possibility 4: The system behaved differently because it was given different input.

Possibility 5: The other possibilities are magnified because your mental model of the system and what influences it is incorrect or incomplete in some important way.

Take a look at the linked blog for suggestions for each type of cause.

It's unpredictable because there is something you don't understand

This answer (and others) list different factors that have been known to create unexpected or apparently unpredictable behavior in software. The software itself can't create non-deterministic random output, full stop. But, it has a lot of inputs, many which are implicit - like the execution time of a thread, network latency, the date and time, is it a leap year, etc.. - which can produce non-deterministic random output. If your software's output is a function of these random inputs, then its output may also be random.

Any test must leave some number of inputs implicit (for practical reasons like time available to write the tests, maintenance concerns, technical limitations). It would be silly to specify the execution time for a function as the input to every test you run (would that ever complicate the test setup!!). But, if controlling/varying the explicit/known inputs can't reproduce the issue, find some implicit/unknown inputs and control/vary those.

The issue then becomes understanding which implicit/unknown inputs affect the behavior:

What we typically call an intermittent problem is: a mysterious and undesirable behavior of a system, observed at least once, that we cannot yet manifest on demand.

Our challenge is to transform the intermittent bug into a regular bug by resolving the mystery surrounding it. After that it’s the programmer’s headache

Your task is to identify what conditions cause that behavior. The linked blog and other answers here give a long list of possible issues. Identify a few that seem likely, and investigate them. Repeat until you can reproduce the bug.

_{Note: I make this point about James Bach's possibilities #1, #2, and #3, and am loosely defining "input" to include the state and the system itself. If you're stuck on that, replace "input" with "input, state, or system characteristics" when you read what I just wrote.}

Perform a binary search on the problem space

When selecting factors to control/vary, choose the ones that give the most bang for your buck by bisecting the problem space.

Example: If your notification emails are intermittently not being received, and you've identified a short list of possible factors as:

1. your domain has been blacklisted
2. there is an issue with a certain mail provider
3. certain code paths aren't triggering notifications
4. spam filters are preventing the mail to go through
5. the user didn't actually take the action that will trigger that notification
6. etc.

To perform a "binary search", you want to do a test that will eliminate the most possible factors at once. In this case maybe that is logging the attempts to send emails. If there is no attempt to send missing emails, then it must be factor 3, 5, or 6. If logging indicates that the missing emails are attempted to be sent, then it must be factor 1, 2, 4, or 6. Repeat with the new problem space. If you find that it's factor 6, generate a new list.

The counter-example is taking "shots in the dark" until you hit it. Example: set up a test that will send notifications to an account from each of the major email providers. If one of them cannot receive notifications, then it must be factor 2 (or maybe factor 4 if it's a spam filter issue with only one provider). Otherwise, it is factor 1, 3, 4, 5, or 6. If you're lucky, this one will take less time. If you're unlucky, much longer.

If you think that your list of known factors (in this example: items 1-5) are less than or equal to the number of unknown factors (in this example: item 6) then a proper first bisection is a test that determines if any of the known factors apply, or if none of them do (in which case you need to make a new list).

TLDR

1. Make a short list of things that might cause it
   • consult the linked article, other answers, google, coworkers, etc.
2. Stuck? Remember: it's not dark magic, it's just something you don't understand yet. See step #1
3. Perform a test that will eliminate half of that list (think: binary search).
4. Repeat as needed.

Answer 3

Analyse from different perspectives to find the exact cause

I am assuming that you are trying to debug the issue in the same environment where it was reported. Few things that should be considered while reproducing the issues:

1. Pile on more logs: Try to capture as much as you can. This is the first thing you can rely on to get some valuable input on such issues.

2. Check the data-related dependencies: Sometimes data is a major contributor in such issues. So, a good approach is to connect to the same database where the issue was reported or create a replica and try to replicate the issue. But, all in all, data aspect should not be ignored.

3. Multi-threaded environment: There are chances that multiple threads update the application data. These can be the cause of unexpected application behavior occasionally.

4. Request more detailed steps from the users reporting the issues:
   There may be some action that the user might find un-related to the issue and don't report, but such action might be import for analysis. Apart form this, you should also ask for data that the users are using for testing (if you have not done already).

5. Try to determine all queries/events that took place during that period.

And I know, it is just like being a sniper and waiting for the issues to re-occur and quite frustrating sometimes.

Answer 4

Apart from the technological aspects, you might need to consider organizational aspects as well.

You will, most likely, have a ticket (a bug report) which remains open/assigned (unresolved) for quite a long time (like several months). In some organizations, this may make you look like a bad performer. In such a case, find a suitable solution with your boss or your bosses' boss to ensure that the "Official KPIs" don't go south. It could be a new status in the ticketing system, like "under long-term investigation", which stops the timers/KPI counters.

Instruct your users that they have to report this bug every time it occurs (or at least once a day, if it occurs more often). This does not need to be the usual detailed bug report, but just a confirmation that the bug is still there. It might happen that the bug disappears at some point in time after some software release which fixes a totally different bug. And you wouldn't know. Or someone could decide that "we have released 10 major versions in the meantime, this bug must have been fixed already" and closes it despite it not being fixed.

And give feedback to your users that you still on the issue, even though you haven't nailed it down yet. Tell then that you appreciate their continued support (and possibly understanding) regarding this issue. Make sure that they consider themselves an important part of the road to the solution, not a nuisance, and that they are taken seriously.

Review your "unsolvable tickets" regularly, so that the issues stay in the back of your mind. It might happen that you come across some piece of code three or six months later, when you are suddenly struck with enlightenment and you suddenly see the obscure side-effect which could trigger such a bug under some obscure and dubious conditions. Obviously, you won't be contentiously hunting for some single bug for weeks or months, but mostly do other work. This keeps you from getting too frustrated. And you view the issue from some different angle after a few weeks of successful and rewarding work on something different.

Discuss the bug with your colleagues and maybe you boss or your user's boss. You colleagues might have some ideas, or stumble across some piece of code three or six months later...you get the idea. The bosses might offer organizational help, like raising user awareness, or even allowing you to capture all network traffic for days or weeks (plus mustering the help from the IT infrastructure department or whatever is needed), or installing monitoring software on the user workstations. Or get the database guys into the boat so they can monitor machine load levels, deadlocks, data corruption (yup), slow queries and such.

You might also check you deployment and build processes (aka Configuration Management). Sometimes bugs are introduced by non-reproducible builds (two builds of the same software version yield different behavior - this can, for example, happen if a build has many manual steps, or builds happen on different machines). Other bugs are caused by some weird side-effect of some other software on the workstations or the servers.

Answer 5

Here is what I instruction my developers to do:

1) add logging; allow the user to select advanced logging. If turning on logging "solves" the problem then it is a timing issue. We add logging to the location in the code that we suspect is involved in the problem the user is seeing.

2) have user take a video of what they are doing when the problem occurs.

3) Run some kind of code analysis tool and resolve issue. There are tools available depending on your language that will analyze the code and give warnings. Resolve as many as you can.

Regarding running a video: I had a problem with an iOS app. The user gave me very detailed instructions on how to reproduce the problem. We got the exact model of iPad and exact iOS and could not reproduce the problem. We asked for a video (they used a different phone to video her using the app).

The user was a very experienced 10 key operator. She hovered her hand over the ipad and would use all five fingers to press the buttons. She was EXTREMELY fast. From that we knew immediately that it was a timing issue.

Answer 6

Here are some ideas:

1. Strongly agree with logging! With time stamps including milliseconds (big difference between events A at 01:23:45.999 and B at 01:23:46.000, vs. A at 01:23:45.000 and B at 01:23:46.999).
2. Add bug-specific logging: if you can detect when a bug has occurred, log stuff.
3. Circular buffers provide low-cost storage for history.
4. Object self-tests for consistent state.
5. Add tracking information to objects: when, and in what context, was this created? Often can be crunched down to a few bytes.
6. Good unit-test coverage.
7. Test automation, including pseudo-random.
8. Thread history: each object has 'checkpoints' that write a 1..2-byte value somewhere that gets logged on failure. Thread A was between checkpoints 123 and 124...
9. Object history: each object has bitwise-ORed flags recording the operations that have occurred (at least once), into a value somewhere that gets logged on failure. Object B got Format() called, but not Init().