1

I'm thinking about new approach to include autotests to CI/CD pipeline and my thought is to determine if a test is flaky before they are in pipeline.

In this case an only way to do that is to

  • setup expected test success rate;
  • launch every test N times (I suppose that would be 100-300) and gather success rate;
  • if test's success rate >= expected success rate, then test can run in CI, if not - test should be fixed or even redesigned.

I think if test has success rate 98% (everyone can choose his own rate-limit) and that test can bring a real profit when launched in CI/CD.

Right now I'm working on the ways to bring that scheme to life and I'd like to check success rate of test everytime new test added or old test changed, so it should be fast (I know some options how to do that, but they still not fast enough and by fast I mean maximum test's run time). First thing that comes to mind is to launch whole my set in parallel mode N times and then gather rates.

But, before I'll start to do that, I'd like to know:

Are there any tools that allows you to check test success rate (or call it flakiness) before test is on the CI pipeline?

2

In similar circumstances I simply used another CI pipeline as the tool, the main reasoning being that re-using of the same execution environment should give the closest results to reality.

You can set up another pipeline with the same settings but running on a single build or even a pre built version so you can focus on the tests part only.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.