SoftwareIt's unpredictable because there is a machinesomething you don't understand
There is some reasonThis answer (and others) list different factors that thehave been known to create unexpected or apparently unpredictable behavior in software is exhibiting. The software itself can't create non-deterministic random output, full stop. But, it has a lot of inputs, many which are implicit - like the behavior thatexecution time of a thread, network latency, the date and time, is it a leap year, etc.. - which can produce non-deterministic random output. If your software's output is a function of these random inputs, then its output may also be random.
Any test must leave some number of inputs implicit (for practical reasons like time available to write the tests, maintenance concerns, technical limitations). It would be silly to specify the execution time for a function as the input to every test you run (would that ever complicate the test setup!!). But, if controlling/varying the explicit/known inputs can't reproduce the issue, find some implicit/unknown inputs and control/vary those.
The issue then becomes understanding which implicit/unknown inputs affect the behavior:
Your task is to identify what conditions cause that behavior. The linked blog and other answers here give a long list of possible issues. Identify a few that seem likely, and investigate them. Repeat until you can reproduce the bug.
Note: I make this point about James Bach's possibilities #1, #2, and #3, and am loosely defining "input" to include the state and the system itself. If you're stuck on that, replace "input" with "input, state, or system characteristics" when you read what I just wrote.
Perform a binary search on the problem space
When selecting factors to control/vary, choose the ones that give the most bang for your buck by bisecting the problem space.
Example: If your notification emails are intermittently not being received, and you've identified a short list of possible factors as:
- your domain has been blacklisted
- there is an issue with a certain mail provider
- certain code paths aren't triggering notifications
- spam filters are preventing the mail to go through
- the user didn't actually take the action that will trigger that notification
- etc.
To perform a "binary search", you want to do a test that will eliminate the most possible factors at once. In this case maybe that is logging the attempts to send emails. If there is no attempt to send missing emails, then it must be factor 3, 5, or 6. If logging indicates that the missing emails are attempted to be sent, then it must be factor 1, 2, 4, or 6. Repeat with the new problem space. If you find that it's factor 6, generate a new list.
The counter-example is taking "shots in the dark" until you hit it. Example: set up a test that will send notifications to an account from each of the major email providers. If one of them cannot receive notifications, then it must be factor 2 (or maybe factor 4 if it's a spam filter issue with only one provider). Otherwise, it is factor 1, 3, 4, 5, or 6. If you're lucky, this one will take less time. If you're unlucky, much longer.
If you think that your list of known factors (in this example: items 1-5) are less than or equal to the number of unknown factors (in this example: item 6) then a proper first bisection is a test that determines if any of the known factors apply, or if none of them do (in which case you need to make a new list).
TLDR
- Make a short list of things that might cause it
- consult the linked article, other answers, google, coworkers, etc.
- Stuck? Remember: it's not dark magic, it's just something you don't understand yet. See step #1
- Perform a test that will eliminate half of that list (think: binary search).
- Repeat as needed.