Our product can integrate with a variety of external systems. We support 5 different databases for storing business data, 5 different version control repositories, 5 different bug trackers,... We also run automated builds nearly every evening (only those where the code changes) with unit tests and integration tests for both our own code and the integration with the external systems we support.
The problem is that for our integration testing, we sometimes run into random, unrepeatable and unpredictable moments where an integration with an external system fails. The most common one we have is a situation where a Torque script to populate a HSQL database fails for no discernible reason, while the one that prompted me to ask this question was a single timeout on an API call to a git repository on a different server in our local network last night. Meanwhile, every other API call to Git and all other systems on that server worked fine. In addition, when we encounter such a problem and we read the failure email sent by our build server when we arrive the next morning, we never see the problem happen during our manual rerun of our automated build, so we don't really know what causes it.
The problem with integration test failures like these is that it's hard to know whether the failure condition is just for a single attempt and as such won't repeat immediately or that it's systemic and you need to fix your integration. Because of that, I don't think automated retries is a viable answer, because if it's a systemic issue, you just made your tests take 5X as long to finish for no real outcome, and if it's an intermittent issue, you're just hiding instability.
Is there any industry standard guidance about a situation like this?