0

I'd like to validate some of my assumptions about what "flakiness" entails in an automated test/suite. To me, that list might include:

  • Non-deterministic (inconsistent pass/fail)
  • Inconsistent run/response times (release related)
  • Test data (invalid, inconsistent across environments, finite use, etc)
  • Retry handlers
  • Dependencies on external services

UPDATE: I'm deliberately excluding environment- and site-specific specific flakiness here (e.g. network latency/access, load, circular dependencies, architecture of test suites), even though, as Cronax pointed out, those are usually the main culprits.

Any thoughts appreciated!

|improve this question
  • 1
    I'd argue that the very things you exclude are the things that usually contribute the most to flakiness. An unreliable environment (often because it's 'just at test environment') is usually the main culprit. – Cronax May 3 '16 at 8:00
  • Agreed. updating question with that feedback, thanks! I guess I'm trying to gauge what other causes there may be if an environment was not an excuse – raven May 3 '16 at 8:27
3

There are number of reasons, why Automated Tests are "flakiness"- Some of them are-

  • Not having a framework
  • Using hardcoded test data
  • Using X,Y coordinates or XPath for element recognition
  • Using shared test environments
  • Having test that are dependent on one another
  • Test not starting in a known state
  • Test no managing their own test data
  • Not treating automation like any other software development effort
  • Failure to use proper synchronization
  • Badly written tests

When running automated tests is to make sure the result of these tests are reliable and consistent. This is especially true when your tests are part of a Continuous Integration system and are run automatically to verify each build. There is nothing worse than a test that passes sometimes and fails others without any new bugs being introduced. These are what are known as “flaky tests”.

How to avoid these flakiness in automated Tests you may check here- How to avoid flakey E2E selenium tests?

|improve this answer
  • Thanks for your post and the link. The majority of your points I interpret as site-specific test architecture issues, which should have fairly straightforward mitigation strategies with proper planning/implementation. By way of example for my question, I'm thinking along the lines of: "our tests are flaky in production but we don't know/understand why" or "a test passed in UAT but failed in production". If it's a non-issue for most, then I'm happy :-) Good point re synchronization by the way – raven May 4 '16 at 9:25
  • Is this answer helpful to you? If you know anythingelse let mw know, otherwise you may mark it as useful or accept the ans .:) – Bharat Mane May 4 '16 at 9:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.