Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

What should I do with tests which flap once every 30 runs or so? This behavior is due to some webdriver or browser issue and cannot be fixed.

Should I delete the test; so that it does not continue to raise false alerts. But in this way, i will also lose the protection against regressions.

What would you do?

share|improve this question
add comment

1 Answer 1

up vote 3 down vote accepted

I assume by "flap" you mean the test is failing unreliably. Unreliable tests generate a lot of noise in the system, and are also costly.

First, I look at the value of this test? Is this test really providing me with valued information every time it runs?

Other considerations include:

  • Is there another way to validate the functionality/behavior at a lower level or with mocks?
  • Automated tests are generally not that smart. Tests should result in a Pass, a Fail, or an Indeterminate condition. If the machine or system is not in the expected state for the test to run properly (e.g. webdriver or browser issue) then your test should log the 'invalid' state and log an indeterminate or skipped result.
share|improve this answer
    
I like your 2nd suggestion: to return a skipped result when the underlying framework behaves incorrectly. –  Asad Iqbal Feb 15 at 1:28
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.