How to avoid flakey E2E selenium tests?

Question

I'm under the impression that most of the E2E selenium tests are flakey. A major reason seems to be timing issues. Another one is that they often test against servers that are not controlled by the test itself (and so are not guaranteed to be available).

Question:

• Have you worked in a project where the E2E selenium were stable?
• How were the flakeyness issues solved?

Answer 1

That is one of the reasons why M.Fowler said "Write as few end-to-end tests as possible". General recommendation is to try to cover as much as possible in your lower level tests (which are faster) e.g. Unit, Integration, Component Tests.

If you have no other options:

• Try to use headless solutions e.g. phantomJS. It may help to have less flakey tests which are failing due to the behaviour of the Graphical Elements in the normal browsers.

• If you found your tests are flakey because of timing issues, try to add/modify desired methods in the Selenium and use them. Example is waitUntillElementIsShown() method.

• Also, "Retry" mechanism can help i.e. re-execute all of the test cases which are failing after each test-suit execution(your Test Automation Job in the CI).

Answer 2

End to end testing will always have a certain amount of flakiness. If you work on a single app or website full time then you start to figure out the characteristics of it and how to deal with them. Timing issues is usually the problem for myself... But getting to know how to use the WebDriverWait class with ExpectedConditions helps dramatically since it lets you make on call reactive wait conditions.

Pay attention to what the DOM is doing during an action, not just the element directly, but the elements around it too. If clicking a button causes the surrounding DOM to do stuff, then creating wait conditions for those other elements as part of your delay might be necessary.

One final suggestion on the topic of timing. Look up how to write your own ExpectedConditions methods. You can write custom tailored conditions to wait on. This gives you a lot of flexibility for those little timing issues you always run into, but can't find a convenient solution prepared by Webdriver.

An example for writing your own custom expected conditions:

    public static ExpectedCondition<Boolean> exampleCondition(final WebElement anElement) {
    return new ExpectedCondition<Boolean>() {

        @Override
        public Boolean apply(WebDriver driver) {
            // you can do something with anElement here.  If false is returned, the condition
            // is not satisfied and this method will be invoked again and again until either
            // true is returned, or the time limit is exceeded.
        }

        @Override
        public String toString() {
            return "I am a condition";
        }
    };
}

Any parameter can be defined as long as it's final, and can be used inside your anonymous implementation.

You can use this like so:

new WebDriverWait(driver, secondsToWait).until(MyExpectedConditions.exampleCondition(someWebElement));