7

I run Web Driver (.Net version) with SpecFlow as my test driver on some fairly general Cross - Browser scenarios so I often am rerunning tests as I add new ones, or clean up old automation. On occasion a test that passed previously suddenly fails for a timeout on an element in a page, or some such similar timeout error, yet running the test 5 minutes before or after gives me a pass. Typically I try to rerun the test right away before looking for issues, considering how often I run the tests I can usually get a pass right away. So I tend to take the stand that I can ignore this error due to some condition beyond my control.

How do others handle this? Should I be taking time t track down intermittent issues which may cause a test to overall give me a failure in <10% of its overall runs, I don't see the value in chasing down one time out when I can rerun the test and see it pass. Or is this something that many people see and I am just trying to find an issue where there is none?

Improve this question

3 Answers 3

Reset to default
9

As a general rule of thumb, everytime a test fails or has an indeterminate outcome it requires investigation. If you ignore a failing/indeterminate test then you increase the risk of missing or overlooking a bug.

The value of automation is to execute tests that we deem important enough that we want to run them repeatitively, or tests that are executed more effectively or efficiently via automated code as compared to manual processes in order to free up our time to do other testing tasks.

If we are spending time baby-sitting tests that are throwing false positives that is time that is being taken away from other testing tasks.

We recently put a lot of effort into improving automated test case reliability. In our critical suites our automated tests must pass a 100x run w/o failure before it is checked-in to the lab. This gives us 99% confidence that any failures in these suites indicate a regression of some sort and are not likely to be false positives.

This was a big cost investment and required in-depth root cause analysis of the different failures we were seeing. 10% may not seem like a lot of effort, but over time and as the # of tests increase over time the costs add up.

Bascially, it boils down to your tipping point of when you are spending (wasting) time investigating false positives versus the time your automated tests should be providing that will free you up to do other testing tasks.

Improve this answer
3
  • 2
    99% confidence: very impressive!
    – user246
    Commented Aug 2, 2012 at 18:43
  • Just to be clear, the 99% confidence is in the test not aborting prematurely or reporting a failure due to reliability issues (but yes, sometimes that does occur). All failures in our critical test suites get high level visibility and folks will stop other work to investigate these failures.
    – Bj Rollison
    Commented Aug 2, 2012 at 21:38
  • I think my issue stems from my tests, being browser only due to the application, intermittently fail somewhere in a chain where I cannot verify the immediate error since a subsequent check passes. In general I have confidence in the Framework, but the individual failures trouble me, but perhaps unnecessarily due to the nature of the Web Framework and Browser. Thanks, this sort of gives me a better frame of mind to look at the failures with.
    – MichaelF
    Commented Aug 6, 2012 at 17:50
1

We also have same issue in our production monitoring automation suite. To solve this problem to some extent, whenever a test is failed it is marked as INTERMITTENT and same test is repeated for max. 3 times or until it is passed. If it is getting passed in any one of next repetition then issue is left as Intermittent and not alerted. If it is failing all the 3 iterations at the same point/step then it is marked as failed and alerted. If it is failing at different steps in different iterations then it is most likely script issue and marked as false positive and not alerted. Hope this helps..

Improve this answer
1
  • This sort of solution is what I need to settle the issues I am having with the failures, I was doing a similar resolution already so it looks like I was not the only one. Thanks!
    – MichaelF
    Commented Aug 7, 2012 at 9:36
0

I also asked question for suggestion how to write reliable test automation, you may find it useful: Your suggestions for writing reliable Web UI automation

Improve this answer
1
  • Actually this is not my problem, I use page objects for running the automation scripts so HTML changes don't really affect my changes. My issue is more focused on the framework having issues where it generates a false negative and how I can handle/deal with that.
    – MichaelF
    Commented Aug 8, 2012 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.