I'd like to validate some of my assumptions about what "flakiness" entails in an automated test/suite. To me, that list might include:
- Non-deterministic (inconsistent pass/fail)
- Inconsistent run/response times (release related)
- Test data (invalid, inconsistent across environments, finite use, etc)
- Retry handlers
- Dependencies on external services
UPDATE: I'm deliberately excluding environment- and site-specific specific flakiness here (e.g. network latency/access, load, circular dependencies, architecture of test suites), even though, as Cronax pointed out, those are usually the main culprits.
Any thoughts appreciated!