Having flaky tests that sometimes fail and sometimes pass is very frustrating:
- there is no single reason each of the flaky tests fail
- the failures don't seem to happen to real users using the application
- it requires a lot of attention and time to dig into the test failures and try to come up with a fix
- they slow the development&testing process down
- they produce false positives - tests that fail without an actual problem but rather because of timing, network, visual effect issues
- it often results in disabling flaky tests in order to move forward because of lack of time
What are the general strategies to handle flaky tests?
If context is needed: we are using Protractor/WebDriverJS for end-to-end browser automatedUI testing.