2

I have a situation where an dependent application is down and I cannot execute the test case. The test execution is automated. The test first calls the dependent application API and this API returns a 500 server error, causing the test to fail. We are using TestNG as test runner.

  • Should my test script fail the test or is it better to skip the test?
  • Should each failed test represent a defect in the Application under test?
Improve this question
2
  • Your post subject and post content are two different questions. What is the question you want to get an anwser for? – Patrick Aug 9 '17 at 8:17
  • I took the liberty to rewrite the question a bit to make it more clear. I think the two questions are close related and grouped it into a new question: How to handle test fails caused by external dependencies. @user2048204 please let us know if I misinterpreted it. – Niels van Reijmersdal Aug 9 '17 at 8:24
4

should it represent a defect in the Application under test?

No, it might be working, but you cannot verify. You do need to monitor and follow it up somehow. Therefore ignoring/skipping the test-method and create a task in your workflow-system might make sense, but this could lead to a lot of waste handling and monitoring these issues. Do make sure you fix the issues as soon as possible, as test not run tend to grow old fast.

It should fail:

Sounds like you are writing end-to-end tests, I would expect this to test the application works end-to-end including external dependencies. Skipping them would give a false sense of security.

If you do not want to be alarmed by a test you know is failing due to external dependencies out of your control I would mute them. The tests should still run, but should not report their failure until either:

  • They run successful once, which removes the mute status
  • The muted time range expired, I would mute tests for some days and then re-evaluate if the mute is still correct.

Our build-server TeamCity supports muting out-of-the-box, for other build-servers you might need to build something yourself. Worst case you add a ignore flag to the test-method and put an item in your agenda to un-ignore them.

Muting best practises:

  • Always use a time range, never mute it forever!
  • Document why it was muted
  • Have cycle where you check muted (also ignored/skipped) tests regularly
  • Plan action to remove the mute
Improve this answer
5
  • +1 Good option (as usual). In the US we would probably use the term 'pause' over mute as mute doesn't suggest restarting at some point and pause does (not 100% but suggests). – Michael Durrant Aug 9 '17 at 8:49
  • @MichaelDurrant I used the terminology that TeamCity uses, I do think that maybe pausing sounds better, but muting is more like snoozing, you do not stop the process you just ignore it temporary. Pausing sounds like stopping it totally. – Niels van Reijmersdal Aug 9 '17 at 8:57
  • No worries. I think of pause as in a performance is paused. To me that implies it will start again. When I think of mute I don't think that of it as something that implies it will "unmute" as some point. "The music was loud so I muted it" seems self contained without the future implication. These may be cultural differences though. I am in north-east US culture currently. I've never heard the term mute in the last 5 years of QE – Michael Durrant Aug 9 '17 at 14:41
  • "Mute" also feels like test continue to be run on the background but any output (failures) is ignored - is it what is happening? And "PASS" will un-mute test automagically. "Paused" test feels like they are waiting to be restarted (they do not progress, do not run at all). Good names are hard. – Peter M. - stands for Monica Aug 9 '17 at 15:25
  • 1
    Yes, that is exactly what is happening. Test keep running and get magically un-muted when they pass again. – Niels van Reijmersdal Aug 9 '17 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.