How to make selenium tests more stable?

Question

I am using selenium tests to perform GUI-based system tests of non-public scientific use cases. Although I have build more-or-less a complete wrapper around the selenium module with functions to always wait for an element to appear (WebDriverWait), to handle inter-nested iframes (switch-to-frame), and even to catch exceptions to try things again (like obtaining the WebDriver in the first place), the tests barely run through fine all the time. I get random failures in different tests all the time, with no obvious cause and not repeatable.

My guess for reasons the tests are failing are:

• selenium is NOT user friendly at all and throws a lot of strange errors without reason
• the pages I load uses many other services that are outside of my control (authentication framework, analysis framework etc)
• the pages load time can be long (10 seconds) and sometimes do not load at all.

Is there maybe a different framework or a kind of wrapper around selenium, which provides methods for SOLID tests that deliver consistent, reliable and reproducible test results? Or are there some other ideas to make selenium tests consistent, reliable and reproducible?

Some of the errors:

• Random Timeout errors (I wait enough time for an element to appear, but it does not appear. Reason currently unknown, very hard to debug because when testing manually these errors do not show up)
• Random WebDriverException: e.g. can't access dead object (again, does not appear when running the single test case manually or automatically).
• Random timeouts on the socket level.
• StaleElementReferenceException
• Element not found although visible, present and clickable and visible on screenshot, and working locally but not inside docker...

Answer 1

I have struggled with this issue in several companies. It always seems to come up at some point and then be a major issue over time.

I too have walked the path of sleeps, polling waits, check for ajax done, check for js done, check for animations done, etc. They sometimes help with some of the errors but it always seems that just when you think you have fixed something and a bunch of runs have passed then wham! Most frustrating of all, when you run the spec yourself it always seems to pass the 2nd time.

My approach to this now is: re-run the example

Again, before resorting to this I put a lot of work into diagnosing the issue and trying various approaches. When various approaches have failed I will consider adding a fixed sleep of 1 to 2 seconds (before polling, i.e. issuing find element commands). If this doesn't help I go for the retry example route.

I am able able to do this easily this with the ruby-rspec-capybara implementation of WebDriver plus use of the rspec-retry gem.

Basically when I've identified one of those intermittent types of errors that I have not been able to resolve, I allow the retry approach to address it.

Initially I only did this for the well known ruby Net::ReadTimeout error. More recently expanded my use to Capybara::ElementNotFound and this is working well.

The main downside is that during test automation development you may wish to disable this so that real element not found situation error out immediately.

This approach is also clearly not ideal and it not favored by some developers. I believe it is a 'real world' compromise and is a good approach given the following caveats:

• The spec usually passes
• The failure rate is low, e.g. 2-5%
• The spec usually passes on re-try
• More specific approaches such as polling waits or ajax waits haven't addressed the issue
• You limit the number of re-tries
• You scope the errors to re-try - this is important, for example I use the array [Net::ReadTimeout, Capybara::ElementNotFound] to limit what gets re-tried.

Answer 2

Flaky test is well-known pain point:

• How to deal with flaky tests that have intermittent failures? (on SO, and links from it)
• How to deal with flaky tests that have intermittent failures? and Where do our flaky tests come from? on Goggle Blog.
• How to fix Flaky Test Automation - another collection, thanks to @Rsf

Because so many extremely smart people were not able to solve it for a long time, I believe flakiness is here to stay, as a inherent part of unreliable world we live in.

Especially last Google Blog link is excellent statistical analysis of where flakiness is coming from (ie. Python WebDriver test are marginally more flaky than Java WebDriver, which might point to unfixed bugs), but their conclusion is that best corelation with flakiness it test size: the longer the test, the bigger chance for it to be flaky.

So:

Against flaky test, even Google themselves content in vain.

Which is of course play on Against stupidity the gods themselves contend in vain - a famous sci-fi novel by Isaac Asimov.

Jenkins is a popular test runner for continuous integration (CI) which runs test repeatedly and shows "average" status of few past runs as "weather": sunny of test are all passing, "cloudy" if mix of pass and fails, and "rainy" if tests are consistently failing. Which is all you need: Check for "cloudy" if you have time, but worry (and fix) the "rainy".

Answer 3

The number one reason for flaky tests is because of flaky websites

If you're trying to run the tests on a completely static webpage, and you're testing locally, then I'm certain a well written test will work 100% of the time.

However, if it:

• uses databases to fill in data,
• uses js/node/ajax to dynamically update the page,
• loads in resources (js/css) from a remote server,
• is hosted on a server not under your direct control,
• or relies on angular/react/whatever framework to add animation and other pretty touches

...you'll need to accept a certain amount of failure.

Any one of those can be a point of failure and/or slowness which will cause tests to become unstable. And considering most websites worth testing are based on most, if not all, of these factors, you are fighting a losing battle.

Answer 4

Short Answer: By asserting states of data & test objects(locators) instead of assuming.

Long Answer:

Automated UI(including selenium) tests can be made more stable by reducing flakiness.

But why Flakiness happens in the first place???

Flakiness happens when things get into undesired & unexpected state. And solution? Don't assume state , assert state(on object & data) before performing any UI operation.

By asserting state be it data or objects, a flaky system/ application can be isolated from test automation.

Before performing any UI operation make sure you are at the expected page,application data is in expected state and the object on which you are performing operation is also in expected state.

Test Data:For data(test data) , I would make sure data is always in known base state before test starts.I generally recommend refreshing/restoring database before run as a part of setup/configuration step.

Locators:For objects(locators), I would assert the pages after loads and other locators states( is Displayed, isEnabled,isClickable etc.)before performing the operation.

Use assertions as safety nets so if something goes wrong it would help you to pinpoint quickly the root cause.

I consider assertions as unit tests for UI test automation.

Answer 5

From the looks of it, the biggest issue is not your Selenium tests but the application (or test environment) itself.

If your tests are written in a robust manner, and they are still showing a lot of errors (such as timeouts on page loads, elements that don't appear in time, ...), then I would advise you: leave the tests as they are, because they proving their value by showing the (performance) issues present.

By updating your tests to work on a buggy application you are helping no one.

Answer 6

Selenium itself does not throw lots of weird errors for no reason. All errors describe what happened and there are lots of suggestions on why what happened and how to fix that when you google them.

If your application under test is unstable (sometimes it doesn't work) - there is no way to improve that with tests, obviously, but you can check what doesn't work and when. Maybe part of page loads really slowly - so it could be a good idea to write a method that verifies some part is there and use that in tests.

If pages load slowly, then add waits for page to be loaded.

But yes, I find using selenium methods without wrapping extremely annoying. I extended all of them similarly like this ( C# ):

 public static string GetAttribute(this IWebElement e, string attr, IWebDriver d, WebDriverWait w)
    {

        Pause(1000);

        var c = WaitInSeconds(w);

        c = c - 2;

        var w2 = new WebDriverWait(d, TimeSpan.FromSeconds(c));

        string att = null;

        if (c <= 1)
        {
            throw new Exception("Could not get attribute.");
        }         

        try
        {
            att = e.GetAttribute(attr);              
        }

        catch (Exception ex)
        {
            Console.WriteLine("\nTried to get attribute, but got an exception: \n" + ex + "\n");
            Pause(500);
            e.GetAttribute(attr, d, w2);

        }

        Console.WriteLine("Got: " +attr+ " value: "+att+ " and counter is at: " +c);

        return att;
    }

And that reduced random stale element, briefly covered by another element, and such, errors dramatically.

Problem is tests fail a lot later. However selenium tests themselves are not fast, and speed can be improved in a lot of different ways, so I think that is good enough trade off to not have to debug every click in every test and catch errors there.

Answer 7

Is there maybe a different framework or a kind of wrapper around selenium, which provides methods for SOLID tests that deliver consistent, reliable and reproducible test results?

Yes. If you work in Java:

• for AJAX applications I recommend you using Selenide. When testing AJAX applications the test often needs to wait until some element changes its state. Selenide has built-in methods for waiting.
• In addition, if you test for AngularJS application, I found NgWebDriver very useful for waiting for Angular to finish async activity. For instance, I had a table with a filter and I wanted to check if an entry exists for specific query I've put in the filter. Entering a query results in updating the table in the background, but you need to wait until it completes before checking the result.

Or are there some other ideas to make selenium tests consistent, reliable and reproducible?

Yes, there are.

Some of the errors:

• Random Timeout errors (I wait enough time for an element to appear, but it does not appear. Reason currently unknown, very hard to debug because when testing manually these errors do not show up)

Those things are pretty hard to reproduce. There can be different reasons behind that, but they might not be related to Selenium per se. I found such problems when testing on different test environments, different browsers (Chrome vs. Firefox) or even different versions of the same browser (Firefox 49 vs. 56).

Random WebDriverException: e.g. can't access dead object (again, does not appear when running the single test case manually or automatically).

This is very generic. It really depends on WebDriverException you have in mind.

Random timeouts on the socket level.

This is very generic. It really depends what socket you mean and perhaps deserves a separate question here.

StaleElementReferenceException

Often Selenide I mentioned above resolves such problems. There's also whole thread How to avoid “StaleElementReferenceException” in Selenium? on SO on how to handle that.

Answer 8

At our company we had about 1000 selenium tests. We ran the tests for every push (as part of a CI process), and what we did to make the tests stable is to simply run the failing tests again, and then again. If after 3 times the same test failed, the build would fail too.