Some background:
We use travis-ci to run our cucumber tests on every commit. Occasionally I am noticing a rather troubling problem. A cucumber test fails, followed by the rest of the tests failing, but yet travis says the build passed. It is simply because travis-ci only cares about exit codes, and for some reason, cucumber is returning an exit code of 0 for this specific error.
Here is what's causing the problem.
- The first bunch of tests work fine, they pass and cucumber thinks they pass
- Then we get a WebKit failure
- No response received from the server. (Capybara::Webkit::NoResponseError)
- Then all tests after give the error: Broken pipe (Errno::EPIPE)
This failure only occasionally happens. The scary part and thing that I actually care about are cucumber returns an exit of 0. Which mean travis-ci doesn't mark it as failed, and deploys my code.
Here is the travis output
70 scenarios (35 failed, 3 skipped, 32 passed) 382 steps (22 failed, 111 skipped, 249 passed) 6m43.713s
The command "$TEST_SUITE" exited with 0.
Has anyone had this problem with cucumber returning the wrong exit code? Is the fault even cucumbers or capybara/webkit? I've tried both irc channels with no response (it is the day after christmas, so not too shocking).
