Some background:

We use travis-ci to run our cucumber tests on every commit. Occasionally I am noticing a rather troubling problem. A cucumber test fails, followed by the rest of the tests failing, but yet travis says the build passed. It is simply because travis-ci only cares about exit codes, and for some reason, cucumber is returning an exit code of 0 for this specific error.

Here is what's causing the problem.

  • The first bunch of tests work fine, they pass and cucumber thinks they pass
  • Then we get a WebKit failure
    • No response received from the server. (Capybara::Webkit::NoResponseError)
  • Then all tests after give the error: Broken pipe (Errno::EPIPE)

This failure only occasionally happens. The scary part and thing that I actually care about are cucumber returns an exit of 0. Which mean travis-ci doesn't mark it as failed, and deploys my code.

Here is the travis output

70 scenarios (35 failed, 3 skipped, 32 passed) 382 steps (22 failed, 111 skipped, 249 passed) 6m43.713s

The command "$TEST_SUITE" exited with 0.

Has anyone had this problem with cucumber returning the wrong exit code? Is the fault even cucumbers or capybara/webkit? I've tried both irc channels with no response (it is the day after christmas, so not too shocking).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.