How should I handle failing tests caused by external dependencies?

Question

I have a situation where an dependent application is down and I cannot execute the test case. The test execution is automated. The test first calls the dependent application API and this API returns a 500 server error, causing the test to fail. We are using TestNG as test runner.

• Should my test script fail the test or is it better to skip the test?
• Should each failed test represent a defect in the Application under test?

Answer 1

should it represent a defect in the Application under test?

No, it might be working, but you cannot verify. You do need to monitor and follow it up somehow. Therefore ignoring/skipping the test-method and create a task in your workflow-system might make sense, but this could lead to a lot of waste handling and monitoring these issues. Do make sure you fix the issues as soon as possible, as test not run tend to grow old fast.

It should fail:

Sounds like you are writing end-to-end tests, I would expect this to test the application works end-to-end including external dependencies. Skipping them would give a false sense of security.

If you do not want to be alarmed by a test you know is failing due to external dependencies out of your control I would mute them. The tests should still run, but should not report their failure until either:

• They run successful once, which removes the mute status
• The muted time range expired, I would mute tests for some days and then re-evaluate if the mute is still correct.

Our build-server TeamCity supports muting out-of-the-box, for other build-servers you might need to build something yourself. Worst case you add a ignore flag to the test-method and put an item in your agenda to un-ignore them.

Muting best practises:

• Always use a time range, never mute it forever!
• Document why it was muted
• Have cycle where you check muted (also ignored/skipped) tests regularly
• Plan action to remove the mute