9

A flaky test is a test which could fail or pass for the same configuration.

What are the plugins that you use to identify and handle flaky tests on Jenkins?

So far I have found the following two

  • Flaky Test Handler Plugin: This plugin is designed to handle flaky tests, including re-running failed tests, aggregate and report flaky tests statistics and so on.
  • Test Results Analyzer Plugin: Displays a matrix subsequent runs of the same tests, so you can identify which tests are ocassionaly red.
Improve this question
2
  • 1
    Is there something in particular you want the plugin to do?
    – K-8
    Oct 12, 2017 at 21:55
  • Instead of relying on a plugin to handle that, I would look for a framework that has the capability out of the box. One such thing I came across is github.com/Codeception/CodeceptJS
    – Barney
    Nov 14, 2017 at 2:39

1 Answer 1

Reset to default
1

Sorry if this is not the answer you want to hear but: Tests should not be flakey

If your test doesn't run to completion then you should consider spitting it up into smaller pieces or changing your orchestration to increase stability. Its more important to maintain confidence in tests than it is to have more coverage.

Having flakey tests can waste a lot of test resources and block build chains from running in CI/CD environments.

The Flakey test plugin can help with this but its really better to quarantine these tests and diagnose the issues because you really can't trust a pass for this.

Some of the main causes of flakey tests are :

  1. Waiting - try to use exception handlers instead of waiting for fixed periods of time as they can finish sooner as well but don't get messed up when a previous delay extends too far.

  2. Not failing - its better to declare a failure when things don't work , in a test you should not be trying to handle the problem (unless you are mocking part of an application that does this specifically and then you should maintain that contract). Its a classic error that people who write code for applications not tests do.

  3. Communication - when you have a flakey test you should communicate this with the developers and get them to help resolve it, usually i do a pair session with developer and we can resolve or at least encapsulate the flakey behaviour. Often this is just because a dependency was not upto date or not delivered in time , this can be hard to find out if you don't keep communicating with the devs.

If you find this situation happens a lot it might be worth setting up 3 amigos meetings to ensure you don't end up working on tasks that are not ready. https://www.agilealliance.org/glossary/three-amigos

Improve this answer
6
  • 1
    Microsft recognizes flakinessis a nature of certain types of tests (with high recall in bug finding): uploads.pnsqc.org/2016/papers/…. They serve different purposes than regular regression tests. You want to remove flakiness in regresion tests that you use for gateway control in your process.
    – dzieciou
    Jan 2, 2018 at 12:58
  • yep and i recognise the flakiness in microsoft products ;-) (bazinga !)
    – Amias
    Jan 2, 2018 at 13:00
  • I think you missed the point without even reading the article.
    – dzieciou
    Jan 2, 2018 at 13:01
  • i read it , i get the point, there is some good insight but thats more about desperately trying to gate quality issues rather than best practice. That document also refers to probably one the biggest messes of software development history and it clearly failed because office is famously buggy mess. Its a clear warning to me, an interesting read tho thanks. There is a way to use flakey tests but the real take away is quarantine flakey tests and don't attempt to fix the failure within the test or test suite.
    – Amias
    Jan 2, 2018 at 13:12
  • Google also recognizes that 1.5% of test results are continuously flaky, even if you have measures to avoid and address them. testing.googleblog.com/2016/05/…
    – Henrik Nordvik
    Mar 13, 2019 at 18:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.