Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Some background:

We use travis-ci to run our cucumber tests on every commit. Occasionally I am noticing a rather troubling problem. A cucumber tests fails, followed by the rest of the tests failing, but yet travis says the build passed. It is simply because travis-ci only cares about exit codes, and for some reason, cucumber is returning an exit code of 0 for this specific error.

Here is what's causing the problem.

  • First bunch of tests work fine, they pass and cucumber thinks they pass
  • Then we get a webkit failure
    • No response received from the server. (Capybara::Webkit::NoResponseError)
  • Then all tests after give the error: Broken pipe (Errno::EPIPE)

This failure only occasionally happens. The scary part and thing that I actually care about, is cucumber returns an exit of 0. Which mean travis-ci doesn't mark it as failed, and deploys my code.

Here is the travis output

70 scenarios (35 failed, 3 skipped, 32 passed) 382 steps (22 failed, 111 skipped, 249 passed) 6m43.713s

The command "$TEST_SUITE" exited with 0.

Has anyone had this problem with cucumber returning the wrong exit code. Is the fault even cucumbers or capybara/webkit? I've tried both irc channels with no response (it is the day after christmas, so not too shocking).

share|improve this question
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.