I'm thinking about new approach to include autotests to CI/CD pipeline and my thought is to determine if a test is flaky before they are in pipeline.
In this case an only way to do that is to
- setup expected test success rate;
- launch every test N times (I suppose that would be 100-300) and gather success rate;
- if test's success rate >= expected success rate, then test can run in CI, if not - test should be fixed or even redesigned.
I think if test has success rate 98% (everyone can choose his own rate-limit) and that test can bring a real profit when launched in CI/CD.
Right now I'm working on the ways to bring that scheme to life and I'd like to check success rate of test everytime new test added or old test changed, so it should be fast (I know some options how to do that, but they still not fast enough and by fast I mean maximum test's run time). First thing that comes to mind is to launch whole my set in parallel mode N times and then gather rates.
But, before I'll start to do that, I'd like to know:
Are there any tools that allows you to check test success rate (or call it flakiness) before test is on the CI pipeline?