"How to Investigate Intermittent Problems" by James Bach provides a very thorough treatment of the subject. Here's an outline of his 92 suggestions:
If the bug exists, it has a cause:
Possibility 1: The system is NOT behaving differently. The apparent intermittence is an artifact of the observation.
Possibility 2: The system behaved differently because it was a different system.
Possibility 3: The system behaved differently because it was in a different state.
Possibility 4: The system behaved differently because it was given different input.
Possibility 5: The other possibilities are magnified because your mental model of the system and what influences it is incorrect or incomplete in some important way.
Take a look at the linked blog for suggestions for each type of cause.
It's unpredictable because there is something you don't understand
This answer (and others) list different factors that have been known to create unexpected or apparently unpredictable behavior in software. The software itself can't create non-deterministic random output, full stop. But, it has a lot of inputs, many which are implicit - like the execution time of a thread, network latency, the date and time, is it a leap year, etc.. - which can produce non-deterministic random output. If your software's output is a function of these random inputs, then its output may also be random.
Any test must leave some number of inputs implicit (for practical reasons like time available to write the tests, maintenance concerns, technical limitations). It would be silly to specify the execution time for a function as the input to every test you run (would that ever complicate the test setup!!). But, if controlling/varying the explicit/known inputs can't reproduce the issue, find some implicit/unknown inputs and control/vary those.
The issue then becomes understanding which implicit/unknown inputs affect the behavior:
What we typically call an intermittent problem is: a mysterious and undesirable behavior of a system, observed at least once, that we cannot yet manifest on demand.
Our challenge is to transform the intermittent bug into a regular bug by resolving the mystery surrounding it. After that it’s the programmer’s headache
Your task is to identify what conditions cause that behavior. The linked blog and other answers here give a long list of possible issues. Identify a few that seem likely, and investigate them. Repeat until you can reproduce the bug.
Note: I make this point about James Bach's possibilities #1, #2, and #3, and am loosely defining "input" to include the state and the system itself. If you're stuck on that, replace "input" with "input, state, or system characteristics" when you read what I just wrote.
Perform a binary search on the problem space
When selecting factors to control/vary, choose the ones that give the most bang for your buck by bisecting the problem space.
If your notification emails are intermittently not being received, and you've identified a short list of possible factors as:
- your domain has been blacklisted
- there is an issue with a certain mail provider
- certain code paths aren't triggering notifications
- spam filters are preventing the mail to go through
- the user didn't actually take the action that will trigger that notification
To perform a "binary search", you want to do a test that will eliminate the most possible factors at once. In this case maybe that is logging the attempts to send emails. If there is no attempt to send missing emails, then it must be factor 3, 5, or 6. If logging indicates that the missing emails are attempted to be sent, then it must be factor 1, 2, 4, or 6. Repeat with the new problem space. If you find that it's factor 6, generate a new list.
The counter-example is taking "shots in the dark" until you hit it. Example: set up a test that will send notifications to an account from each of the major email providers. If one of them cannot receive notifications, then it must be factor 2 (or maybe factor 4 if it's a spam filter issue with only one provider). Otherwise, it is factor 1, 3, 4, 5, or 6. If you're lucky, this one will take less time. If you're unlucky, much longer.
If you think that your list of known factors (in this example: items 1-5) are less than or equal to the number of unknown factors (in this example: item 6) then a proper first bisection is a test that determines if any of the known factors apply, or if none of them do (in which case you need to make a new list).
- Make a short list of things that might cause it
- Stuck? Remember: it's not dark magic, it's just something you don't understand yet. See step #1
- Perform a test that will eliminate half of that list (think: binary search).
- Repeat as needed.