Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Some background:

We use travis-ci to run our cucumber tests on every commit. Occasionally I am noticing a rather troubling problem. A cucumber tests fails, followed by the rest of the tests failing, but yet travis says the build passed. It is simply because travis-ci only cares about exit codes, and for some reason, cucumber is returning an exit code of 0 for this specific error.

Here is what's causing the problem.

  • First bunch of tests work fine, they pass and cucumber thinks they pass
  • Then we get a webkit failure
    • No response received from the server. (Capybara::Webkit::NoResponseError)
  • Then all tests after give the error: Broken pipe (Errno::EPIPE)

This failure only occasionally happens. The scary part and thing that I actually care about, is cucumber returns an exit of 0. Which mean travis-ci doesn't mark it as failed, and deploys my code.

Here is the travis output

70 scenarios (35 failed, 3 skipped, 32 passed) 382 steps (22 failed, 111 skipped, 249 passed) 6m43.713s

The command "$TEST_SUITE" exited with 0.

Has anyone had this problem with cucumber returning the wrong exit code. Is the fault even cucumbers or capybara/webkit? I've tried both irc channels with no response (it is the day after christmas, so not too shocking).

share|improve this question

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.