Follow-up to How to deal with flaky tests?
One of the approaches to tackle flaky tests is to temporarily quarantine the tests - move them to a special "bucket" with other flaky tests:
I offer one piece of essential advice: quarantine your non-deterministic tests. If you have non-deterministic tests keep them in a different test suite to your healthy tests. That way you'll you can continue to pay attention to what's going on with your healthy tests and get good feedback from them.
How would you actually implement this quarantine?
We use Protractor/WebDriverJS, tests are written in Jasmine, we use Jenkins/Git for build&test.
Ideally this bucket needs to have the following properties:
- tests in the quarantine are not executed during a regular build&test run
- do not allow to have more than N tests there (if you want to add a test to quarantine and it is full - you have to first fix one of them)
- do not allow the tests rot in the quarantine
suitewith "flaky" tests
- make these tests "pending" via
- add a "tag" to a test description
I'm though not sure how to enforce the properties listed above.
pendingmay be all you need.