I am having a recurring problem where bugs are cropping up once or twice in a build, and are reported by users but I and my team are not able to replicate them.

What techniques do you use to replicate bugs that are hard to replicate?

We are using Sentry which picks up exceptions but not glitches, which are the most frustrating bugs to solve since they appear once and reappear infrequently.

  • 1
    Please define "glitches" – Michael Durrant Jun 29 '17 at 9:31
  • 1
    Feature bugs, then works 100 times, bugs up again. – bobo2000 Jun 29 '17 at 10:20
  • 6
    You need to impress on your users the importance of reproducibility, unless they can describe the steps needed to expose the bug, there is little or nothing you can do about it. Don't be drawn into the trap of saying I'll take a look. – Martin Spamer Jun 29 '17 at 11:01
  • 1
    The problem is that they do give us the steps to replicate, but then when we try it, it does not break. – bobo2000 Jun 29 '17 at 11:03
  • 6
    This sounds like an initialization error in the underlying code. The error the user sees is caused not by their actions, but by the machine state at the time the failure occurs. Instead of trying to mimic users' actions, have the programmers reverse engineer the root cause from the exception trace. If I am correct and this is an initialization error, then no one will be able to reproduce the symptoms on demand, especially not in a debugging or sandboxed testing environment. – pojo-guy Jun 29 '17 at 11:43

You have

intermittent failures

Welcome to the crowd. In my experience they are the norm and the bane of Quality Engineering.
So first off - accept that and start making resources available for it.

Some techniques that you can use:

  • Allow sufficient resources. Be prepared to need to take days/weeks to fix intermittent failures
  • Collect examples. Some intermittent failures don't reveal patterns until I see dozens of them
  • Run a test locally multiple times. At least 100, if not 1000 times. Look for patterns
  • Review the test setups. Test should not be sharing any data, even reference data
  • For tests, use implicit wait for elements not explicit waits (sleeps)
  • Use and enhance logs to show more info for specific incidents
  • Run a test 100 or 1000 times on your ci server

Useful information for investigations that I have collected over time:

  • Browser
  • User type
  • Use of ajax
  • Time of day
  • Screen size
  • Browser version
  • Use of javascript
  • Frequency over time
  • Various resource ids
  • Function being performed
  • Relation to release pushes / reboots
  • 1
    You observe and collect data on the tests that occasionally fail. Sometimes it takes months or even years to see the pattern behind the failure. – Michael Durrant Jun 29 '17 at 10:53
  • 7
    It's not quite the same, but I remember 30 years ago, a developer could not reproduce a crash that a user was reporting (the user could fairly easily produce it). It turned out that the difference was that the developer had an odd number of characters in their user name, and the user had an even number; that was enough to make the difference between triggering the bug and not. – Martin Bonner Jun 29 '17 at 13:18
  • 1
    Martin Bonner's anecdote adds another approach to track down such problems: one by one, compare the user's and your application environment - from different code or build versions, external libraries, environment variables, OS restrictions (user access etc.) until you find a difference. Unfortunately, there are probably lots of them. Reproduce the environment, then test again. – not2savvy Jun 29 '17 at 13:27
  • 5
    @MartinBonner If user can reproduce the bug easily, it is NOT intermittent bug. It is just a bug for which we don't know (yet) the triggers. Really intermittent bug is intermittent also for the user who reported it. – P.M Jun 29 '17 at 16:39
  • 2
    @bobo2000 Depends on what you mean by "not know how to test". If by "test" you mean "run a short bit of code and get a clear, definitive fail signal", you're overly optimistic. Instead, you're "testing" to find the failure, not debug it. From the bug description you know the general area that's giving issues. Start banging on it. Do the thing that supposedly isn't working hundreds of times. Do hundreds of variations of the thing that isn't working. Use differences between environments where it is working and where it isn't to form hundreds of hypotheses and test each of those. – R.M. Jun 29 '17 at 17:54

"How to Investigate Intermittent Problems" by James Bach provides a very thorough treatment of the subject. Here's an outline of his 92 suggestions:

If the bug exists, it has a cause:

Possibility 1: The system is NOT behaving differently. The apparent intermittence is an artifact of the observation.

Possibility 2: The system behaved differently because it was a different system.

Possibility 3: The system behaved differently because it was in a different state.

Possibility 4: The system behaved differently because it was given different input.

Possibility 5: The other possibilities are magnified because your mental model of the system and what influences it is incorrect or incomplete in some important way.

Take a look at the linked blog for suggestions for each type of cause.

It's unpredictable because there is something you don't understand

This answer (and others) list different factors that have been known to create unexpected or apparently unpredictable behavior in software. The software itself can't create non-deterministic random output, full stop. But, it has a lot of inputs, many which are implicit - like the execution time of a thread, network latency, the date and time, is it a leap year, etc.. - which can produce non-deterministic random output. If your software's output is a function of these random inputs, then its output may also be random.

Any test must leave some number of inputs implicit (for practical reasons like time available to write the tests, maintenance concerns, technical limitations). It would be silly to specify the execution time for a function as the input to every test you run (would that ever complicate the test setup!!). But, if controlling/varying the explicit/known inputs can't reproduce the issue, find some implicit/unknown inputs and control/vary those.

The issue then becomes understanding which implicit/unknown inputs affect the behavior:

What we typically call an intermittent problem is: a mysterious and undesirable behavior of a system, observed at least once, that we cannot yet manifest on demand.

Our challenge is to transform the intermittent bug into a regular bug by resolving the mystery surrounding it. After that it’s the programmer’s headache

Your task is to identify what conditions cause that behavior. The linked blog and other answers here give a long list of possible issues. Identify a few that seem likely, and investigate them. Repeat until you can reproduce the bug.

Note: I make this point about James Bach's possibilities #1, #2, and #3, and am loosely defining "input" to include the state and the system itself. If you're stuck on that, replace "input" with "input, state, or system characteristics" when you read what I just wrote.

Perform a binary search on the problem space

When selecting factors to control/vary, choose the ones that give the most bang for your buck by bisecting the problem space.

Example: If your notification emails are intermittently not being received, and you've identified a short list of possible factors as:

  1. your domain has been blacklisted
  2. there is an issue with a certain mail provider
  3. certain code paths aren't triggering notifications
  4. spam filters are preventing the mail to go through
  5. the user didn't actually take the action that will trigger that notification
  6. etc.

To perform a "binary search", you want to do a test that will eliminate the most possible factors at once. In this case maybe that is logging the attempts to send emails. If there is no attempt to send missing emails, then it must be factor 3, 5, or 6. If logging indicates that the missing emails are attempted to be sent, then it must be factor 1, 2, 4, or 6. Repeat with the new problem space. If you find that it's factor 6, generate a new list.

The counter-example is taking "shots in the dark" until you hit it. Example: set up a test that will send notifications to an account from each of the major email providers. If one of them cannot receive notifications, then it must be factor 2 (or maybe factor 4 if it's a spam filter issue with only one provider). Otherwise, it is factor 1, 3, 4, 5, or 6. If you're lucky, this one will take less time. If you're unlucky, much longer.

If you think that your list of known factors (in this example: items 1-5) are less than or equal to the number of unknown factors (in this example: item 6) then a proper first bisection is a test that determines if any of the known factors apply, or if none of them do (in which case you need to make a new list).

TLDR

  1. Make a short list of things that might cause it
  2. Stuck? Remember: it's not dark magic, it's just something you don't understand yet. See step #1
  3. Perform a test that will eliminate half of that list (think: binary search).
  4. Repeat as needed.
  • 1
    Even with a late start, your answer is gaining ground, and you are gaining XP. Not bad for a first answer. Welcome on board! Bring on those links and answers! – P.M Jun 29 '17 at 18:46
  • 4
    "there is no such thing as random behavior in software" - this is simply not true, all sorts of software exhibit random behavior, even when not designed to be random. Anything multithreaded/async/distributed is usually nondeterministic. Disk and network latency are random factors. – Arcinde Jun 29 '17 at 23:33
  • 2
    @Arcinde, in practice I agree with you. But pedantically speaking, if the source of randomness is coming from some hardware like a disk, then I would argue that it's acting as a type of hardware random number generator. On the other hand, if it's coming from software complexity as in a multithreaded/async/distributed system, then strictly speaking it's not random - nondeterministic perhaps, but not random. Expand the system boundaries wide enough and the nondeterministic system becomes deterministic. That's my point. – alexanderbird Jun 29 '17 at 23:51
  • 1
    "nondeterministic perhaps, but not random" - I have to disagree. On a modern processor there's so much going on that even just "how many clock cycles does it take" is often truly random, not just infeasible. Asynchronous signals on those timescales have too much influence from shot noise / EMI / etc / etc to be anywhere near deterministic. And there are more asynchronous signals on a modern processor than one might think. – TLW Jun 30 '17 at 4:49
  • 1
    Rather than "random", chaotic behavior might be a better choice of words. Way back in 1975 a mathematical proof was offered that any system with three independent variables would exhibit chaotic behavior. While chaotic is not random, there is no predicting which state the system will be in (Period three implies chaos, James A York) – pojo-guy Jul 11 '17 at 4:46

Analyse from different perspectives to find the exact cause

I am assuming that you are trying to debug the issue in the same environment where it was reported. Few things that should be considered while reproducing the issues:

  1. Pile on more logs: Try to capture as much as you can. This is the first thing you can rely on to get some valuable input on such issues.

  2. Check the data-related dependencies: Sometimes data is a major contributor in such issues. So, a good approach is to connect to the same database where the issue was reported or create a replica and try to replicate the issue. But, all in all, data aspect should not be ignored.

  3. Multi-threaded environment: There are chances that multiple threads update the application data. These can be the cause of unexpected application behavior occasionally.

  4. Request more detailed steps from the users reporting the issues:
    There may be some action that the user might find un-related to the issue and don't report, but such action might be import for analysis. Apart form this, you should also ask for data that the users are using for testing (if you have not done already).

  5. Try to determine all queries/events that took place during that period.

And I know, it is just like being a sniper and waiting for the issues to re-occur and quite frustrating sometimes.

  • What type of logs do you recommend? – bobo2000 Jun 30 '17 at 10:05
  • It's a web app we are building – bobo2000 Jun 30 '17 at 10:05
  • You should go for both server side and client side logging. I mean it also depends on the type of issues. If the issues are related to UI then client side logging might be enough. In case of functional/ data related issues, server side logging is recommended. And just write simple logs which enable you to record information after every logical step. – Aalok Jul 3 '17 at 10:45

Apart from the technological aspects, you might need to consider organizational aspects as well.

You will, most likely, have a ticket (a bug report) which remains open/assigned (unresolved) for quite a long time (like several months). In some organizations, this may make you look like a bad performer. In such a case, find a suitable solution with your boss or your bosses' boss to ensure that the "Official KPIs" don't go south. It could be a new status in the ticketing system, like "under long-term investigation", which stops the timers/KPI counters.

Instruct your users that they have to report this bug every time it occurs (or at least once a day, if it occurs more often). This does not need to be the usual detailed bug report, but just a confirmation that the bug is still there. It might happen that the bug disappears at some point in time after some software release which fixes a totally different bug. And you wouldn't know. Or someone could decide that "we have released 10 major versions in the meantime, this bug must have been fixed already" and closes it despite it not being fixed.

And give feedback to your users that you still on the issue, even though you haven't nailed it down yet. Tell then that you appreciate their continued support (and possibly understanding) regarding this issue. Make sure that they consider themselves an important part of the road to the solution, not a nuisance, and that they are taken seriously.

Review your "unsolvable tickets" regularly, so that the issues stay in the back of your mind. It might happen that you come across some piece of code three or six months later, when you are suddenly struck with enlightenment and you suddenly see the obscure side-effect which could trigger such a bug under some obscure and dubious conditions. Obviously, you won't be contentiously hunting for some single bug for weeks or months, but mostly do other work. This keeps you from getting too frustrated. And you view the issue from some different angle after a few weeks of successful and rewarding work on something different.

Discuss the bug with your colleagues and maybe you boss or your user's boss. You colleagues might have some ideas, or stumble across some piece of code three or six months later...you get the idea. The bosses might offer organizational help, like raising user awareness, or even allowing you to capture all network traffic for days or weeks (plus mustering the help from the IT infrastructure department or whatever is needed), or installing monitoring software on the user workstations. Or get the database guys into the boat so they can monitor machine load levels, deadlocks, data corruption (yup), slow queries and such.

You might also check you deployment and build processes (aka Configuration Management). Sometimes bugs are introduced by non-reproducible builds (two builds of the same software version yield different behavior - this can, for example, happen if a build has many manual steps, or builds happen on different machines). Other bugs are caused by some weird side-effect of some other software on the workstations or the servers.

Here is what I instruction my developers to do:

1) add logging; allow the user to select advanced logging. If turning on logging "solves" the problem then it is a timing issue. We add logging to the location in the code that we suspect is involved in the problem the user is seeing.

2) have user take a video of what they are doing when the problem occurs.

3) Run some kind of code analysis tool and resolve issue. There are tools available depending on your language that will analyze the code and give warnings. Resolve as many as you can.

Regarding running a video: I had a problem with an iOS app. The user gave me very detailed instructions on how to reproduce the problem. We got the exact model of iPad and exact iOS and could not reproduce the problem. We asked for a video (they used a different phone to video her using the app).

The user was a very experienced 10 key operator. She hovered her hand over the ipad and would use all five fingers to press the buttons. She was EXTREMELY fast. From that we knew immediately that it was a timing issue.

  • 1
    "add logging" - one technique is to add detailed, automatic logging to a scratch file while the app is running. Then give the user a command somewhere in the apps, UI to "generate crash log" to copy it to somewhere accessible immediately after the crash, and append it to the bug report if the QA team request it. That way, you don't have to rely on what the user thinks (or half-remembers) what he/she did! – alephzero Jun 30 '17 at 1:34
  • What type of logs? – bobo2000 Jun 30 '17 at 10:04

Here are some ideas:

  1. Strongly agree with logging! With time stamps including milliseconds (big difference between events A at 01:23:45.999 and B at 01:23:46.000, vs. A at 01:23:45.000 and B at 01:23:46.999).
  2. Add bug-specific logging: if you can detect when a bug has occurred, log stuff.
  3. Circular buffers provide low-cost storage for history.
  4. Object self-tests for consistent state.
  5. Add tracking information to objects: when, and in what context, was this created? Often can be crunched down to a few bytes.
  6. Good unit-test coverage.
  7. Test automation, including pseudo-random.
  8. Thread history: each object has 'checkpoints' that write a 1..2-byte value somewhere that gets logged on failure. Thread A was between checkpoints 123 and 124...
  9. Object history: each object has bitwise-ORed flags recording the operations that have occurred (at least once), into a value somewhere that gets logged on failure. Object B got Format() called, but not Init().

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.