I'm thinking about a new approach to include autotests to CI/CD pipeline and my thought is to determine if a test is flaky before they are in pipeline.

In this case, the only way to do that is to

  • setup expected test success rate;
  • launch every test N times (I suppose that would be 100-300) and gather success rate;
  • if test's success rate >= expected success rate, then the test can run in CI, if not - the test should be fixed or even redesigned.

I think if the test has a success rate of 98% (everyone can choose his own rate-limit) and that test can bring a real profit when launched in CI/CD.

Right now I'm working on the ways to bring that scheme to life and I'd like to check the success rate of the test every time new test added or old test changed, so it should be fast (I know some options how to do that, but they still not fast enough and by fast I mean maximum test's run time). The first thing that comes to mind is to launch the whole set in parallel mode N times and then gather rates.

But, before I'll start to do that, I'd like to know:

Are there any tools that allow you to check the test success rate (or call it flakiness) before the test is on the CI pipeline?

2 Answers 2


In similar circumstances I simply used another CI pipeline as the tool, the main reasoning being that re-using of the same execution environment should give the closest results to reality.

You can set up another pipeline with the same settings but running on a single build or even a pre built version so you can focus on the tests part only.


DeFlaker is a tool that identifies flaky tests. It is integrated into the build as Maven extension.

It was developed in academia by Bell et al. I have not used the tool myself. Therefore, I cannot make any statements about how mature it is and whether it scales for real-world applications.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.