When nightly testing integrations with external systems, how should random failures of the external system be handled?

Question

Our product can integrate with a variety of external systems. We support 5 different databases for storing business data, 5 different version control repositories, 5 different bug trackers,... We also run automated builds nearly every evening (only those where the code changes) with unit tests and integration tests for both our own code and the integration with the external systems we support.

The problem is that for our integration testing, we sometimes run into random, unrepeatable and unpredictable moments where an integration with an external system fails. The most common one we have is a situation where a Torque script to populate a HSQL database fails for no discernible reason, while the one that prompted me to ask this question was a single timeout on an API call to a git repository on a different server in our local network last night. Meanwhile, every other API call to Git and all other systems on that server worked fine. In addition, when we encounter such a problem and we read the failure email sent by our build server when we arrive the next morning, we never see the problem happen during our manual rerun of our automated build, so we don't really know what causes it.

The problem with integration test failures like these is that it's hard to know whether the failure condition is just for a single attempt and as such won't repeat immediately or that it's systemic and you need to fix your integration. Because of that, I don't think automated retries is a viable answer, because if it's a systemic issue, you just made your tests take 5X as long to finish for no real outcome, and if it's an intermittent issue, you're just hiding instability.

Is there any industry standard guidance about a situation like this?

Answer 1

Is there any industry standard guidance about a situation like this?

Not that I know of or like, but..

for no discernible reason

Unless you use randomness in your environment there is no such thing as un-explainable events. I can accept unrepeatable and unpredictable, but not totally random things.

I can think of (at least) three approaches to overcome your problem, let's call it test flakiness for the sake of this discussion.

1. Well actually there is an industry standard guidance for that- repeat the test until it passes, or at least repeat N times and choose.

   This approach is easy to implement but has no real advantage otherwise, you ignore failures that could be real failures and you waste resources running tests for no good reason.

2. Check the logs, and if you don't find an answer collect more logs and use more intelligence interpreting them.

   Like I wrote above failures are not really random (unless you count Gamma rays flipping memory bits).

   In most systems you can have logs that includes hints about the source of the problem, but you need to know what to look for or enable it if it doesn't exist.

   Sometimes you need to enable more logging, or more fine grained log levels and sometimes you need to add logging to existing software. It's not always a straight forward process to identify what the problem is and what needed to be enabled, sometimes you need to cross information from different sources and sometimes you will need to use machine learning and AI to analyze your files if you want to have results without manual intervention.

3. Fix the root cause of the problem, real life systems are not suppose to be flaky. Read this for example but fixing the problem should be done on all levels from the product itself, through the test environment the the test code.

Answer 2

In situations like this i have tended towards trying to perform a basic health check on the connection before using it. I then attempt to capture relevant information that might explain later errors and ensure its in the logs. I've had a few situations like this and eventually with a enough (or just different) logging you find the tiny detail that explains it.

It sounds like your application is not fault tolerant enough to deal with the environment its running in those errors would be annoying for users. I would do some exploratory testing paired with one of the developers. This way you could look at finding better ways to handle the errors in the first place.

If you can't fix the problem you could at least improve the reliability of your tests by splitting the integration at these failure points, thus allowing each section to be tested. The second half would then run on either synthetic or stored data that should be supplied to test code by the test framework as a parameter. You could then contract test across that split to ensure that both sides are compatible.

Answer 3

I'd look at what you're expecting the tests to tell you, and what you do in response to the failures, to decide what you should do.

If all you're doing initially is manually rerunning the tests the next day, then automatically rerunning them, possibly after a delay, is probably the right thing to do, and you might even want to suppress the email you get about an initial failure, because a singular failure that doesn't repeat isn't useful information to you (I'd still log the failure, though, to look at trends/repeated failures over time).

If you're concerned about a systemic service failure causing an extended runtime, you could have a wrapper for each set of test cases that either tracks failures and aborts the set after a certain number, or does a healthcheck of the service at the beginning, and only allows the set of test cases to run if the service appears to be running properly. Plus, at some level, the whole point of running automated tests off-shift is that you're using otherwise idle capacity to learn something; if the tests take five hours instead of one, but no one's around either way, who cares?

As for logging the failures failing, that's a problem in and of itself. If you can't use the logs to help you debug problems, I'd ask what the point of keeping the logs is. What would have happened if you'd seen a similar failure in production?

If the failures bother you enough, I'd look at mocking the services, although that's probably too complicated to warrant doing. Again, it's a question of what you're trying to test/learn. Given that you don't have log data to tell you what's going on, a failure that involves both your code and the service code isn't debuggable, so intermittent failures aren't useful to you, unless you can somehow determine where the problem is.