What should I do with test flappers which cannot be fixed?

Question

What should I do with tests which flap once every 30 runs or so? This behavior is due to some webdriver or browser issue and cannot be fixed.

Should I delete the test; so that it does not continue to raise false alerts. But in this way, i will also lose the protection against regressions.

What would you do?

Answer 1

I assume by "flap" you mean the test is failing unreliably. Unreliable tests generate a lot of noise in the system, and are also costly.

First, I look at the value of this test? Is this test really providing me with valued information every time it runs?

Other considerations include:

• Is there another way to validate the functionality/behavior at a lower level or with mocks?
• Automated tests are generally not that smart. Tests should result in a Pass, a Fail, or an Indeterminate condition. If the machine or system is not in the expected state for the test to run properly (e.g. webdriver or browser issue) then your test should log the 'invalid' state and log an indeterminate or skipped result.