7

Follow-up to How to deal with flaky tests?

One of the approaches to tackle flaky tests is to temporarily quarantine the tests - move them to a special "bucket" with other flaky tests:

I offer one piece of essential advice: quarantine your non-deterministic tests. If you have non-deterministic tests keep them in a different test suite to your healthy tests. That way you'll you can continue to pay attention to what's going on with your healthy tests and get good feedback from them.

How would you actually implement this quarantine?

We use Protractor/WebDriverJS, tests are written in Jasmine, we use Jenkins/Git for build&test.

Ideally this bucket needs to have the following properties:

  • tests in the quarantine are not executed during a regular build&test run
  • do not allow to have more than N tests there (if you want to add a test to quarantine and it is full - you have to first fix one of them)
  • do not allow the tests rot in the quarantine

Current thoughts:

  • special flaky test suite with "flaky" tests
  • make these tests "pending" via pending jasmine keyword
  • add a "tag" to a test description

I'm though not sure how to enforce the properties listed above.

Improve this question
3
  • 1
    Couldn't you define the test suite, then just write a shell script to grep through the test files and fail the Jenkins build if there are too many tests in this suite, or if any of them have been there too long? Even if you couldn't represent all of this using Jasmine or Protractor directly (I haven't played with them for a few years) you could at least add some comments with the necessary metadata to each "flaky" test. For that matter, if you aren't actually running them there may not be a reason to create a separate test suite--metadata comments and pending may be all you need.
    – c32hedge
    Aug 4, 2017 at 17:15
  • @c32hedge I see your point - continue doing what we are doing now - e.g. making tests "pending", but solve the problem statically and probably externally - analyze how many of the pending tests are there and throw errors if more than a specified threshold. Makes sense, that's definitely an option! Thanks! (feel free to add it as an answer - seems like a valid thing to post to me)
    – alecxe
    Aug 4, 2017 at 17:31
  • Thanks, added answer. Wanted to vet it as a comment first to make sure I understood your question.
    – c32hedge
    Aug 4, 2017 at 17:39

1 Answer 1

Reset to default
1

I'm not familiar enough with Jasmine or Protractor to post a competent answer of how to solve this with them, but here's an external option:

Write a shell script to grep through the test files and fail the Jenkins build if there are too many tests you have designated as "flaky", or if any of them have been marked as such for too long. If Jasmine and Protractor don't give you enough keywords/features to incorporate all the metadata that script would need, just add comments with the necessary metadata to each "flaky" test. For that matter, unless you're actually running the flaky tests there may not be a reason to create a separate test suite for them. Metadata comments, the pending keyword you mentioned, and a script may be all you need.

Improve this answer
1
  • 1
    Good out-of-the-box option! We can may be even make a grunt task out of it and then execute as a part of daily build&test failing the build if quarantine is full. Thanks!
    – alecxe
    Aug 4, 2017 at 17:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.