What techniques can I use to determine the reasons for intermittent automated UI test failures?

Question

Intermittent failures of UI feature specs that run in browsers but fail intermittently are the bane of automated testers lives and I've experienced them at every place I've worked at.

Not so much https://sqa.stackexchange.com/a/3582/8992 but more about what approaches and techniques can I use to first find the underlying issues that are causing them?

Answer 1

My personal opinions:

• Trying to analyse the testing environment closely, identify as many variables that may have effect on testing results as possible
• Invite fresh eyes to aid you during this process of variable identification, but please do this after you have done it so you can compare your results against theirs
• Trying to have a very controlled testing environment, ideally have control over all variables you have identified, this include testing inputs and testing environment variables, such as temperature, timing and hardware.
• Run tests and log everything for future reference
• Purposely alter controlled-variable one at a time, see if it makes any difference to test results, statistically, you may need to do this multiple times
• Statistically category testing failures
• Encourage people to report all intermittent test failures, investigate their reports but do not fully trust their reports as they may leave out detailed information
• When you have found some leads, set a trap; e.g. you suspect hardware version may be the issue, set up a testing farm that consists of the suspected hardware version and innocent hardware version, compare run results statically.

Answer 2

screenshot on failure. I had a problem where I was actually on a completely different page than I was expecting because of overlapping elements when I clicked a link. A screenshot can give really good information as to what is actually happening.