Why are Selenium tests so unstable?

Question

I am running a few scripts (in my personal and business laptop) and every day I get a different error at a different step.

Sometimes I run and it is passed, but I run again and get an error that an element is not visible or so.

I've included implicit waits, explicit waits, also time for page to load, however it doesn't help. How can I fix this?

Answer 1

Main observations in unstable scripts are as follows:

1. Selenium tests are usually unstable because of the unnecessary complexity created in your test framework
2. Enabling parallel execution without keeping in mind that one test may affect the state of the system that in turn affect the other test running in parallel.
3. Using deprecated modules
4. Incorrect configuration of modules that are used in the framework
5. Not using explicit wait for asynchronous actions
6. Using tools that are not meant for testing the software under test (e.g., use protractor for angular webpages than selenium)
7. Developing test in one OS and running it in another OS, in CI/CD test agents may have different OS and configurations than what is used in the development PC

There are many such reasons. Also, whenever investigating such issue, follow the below steps in the order mentioned:

1. Trust your code and doubt software under test (SUT):

If everything was working fine and the test starts to fail suddenly. Instead of debugging your code for issues, start with checking the actual product. Do some visual inspection and see whether the development team has modified the element or the element is no longer being displayed.

2. Trust your code and doubt the environment:

If everything was working fine locally and failed as soon as you integrated to CI/CD. Then investigate the product behaviour in the test server. Mostly due to OS and configuration differences the product won't work as it would in local (raise a bug)

3. Now doubt your scripts (using absolute XPATH):

You might be using an absolute XPATH. This causes flaky tests when the DOM structure changes. Use a relative XPATH (CSS would be more recommended). Never use xpath/ if you have a unique ID/name to identify an element.

4. Now doubt your scripts (not using explicit wait):

Sometimes scripts lack explicit wait and try to interact with dynamic elements, and this causes the test to fail, because it tries to interact with the element before it is even available in the DOM.

5. Now doubt your scripts (handling spinners):

Sometimes spinner takes time to get displayed. So, if you are just checking for the invisibility condition then it will return true and try to interact with the next element before the actual spinner event completes.

So, first, check the visibility of spinner and then check for invisibility before interacting with other dynamic elements.

6. Now doubt your scripts (not handling iFrames):

Sometimes an element will be inside iframes and scripts won't switch between frames before interacting with these elements.

Check whether any parent element contains the tag frame or iframe to determine whether the element is inside an iframe.

Answer 2

The more tests are integrated, the less stable they are. This is because they involve many more environmental factors.

Basically, Selenium tests are not just integrated in terms of software components which compose your AUT, but also in terms of the particular version of the OS, particular browser version, particular graphical driver version, or "headless" display implementation...

They are also "integrated" with numerous libraries which all have their own defects and other specific, with particular networking infrastructure which might cause routing or messaging issues, with your hardware that might not be just powerful enough for rendering the components in a web browser in stable and reasonable time.

So to make your tests stable, you have to stabilize all those factors. If this issue is really critical for you, it is probably better to organize your test execution in virtualized environments where you would be able to control them.

Answer 3

You may be running synchronous tasks against an asynchronous device - a web browser.

Modern browsers, frameworks and pages frequently make calls, such as to API's, vendors, etc. that are done asynchronously. You tests may not account for this behavior and if not will consequently fail intermittently because of it, combined with other factors such as detailed by Alexey.

Answer 4

Great question! Creating and debugging a stable Selenium test can take 1–2 days. In addition to the excellent responses in this thread, if you are using Selenium today, then you no doubt run into errors like these every time there is the slightest code change:

-NoSuchElementException -ElementNotVisibleException -TimeoutException -NoSuchAttributeException

This is because Selenium uses single selectors to identify elements and if they change because of feature or code updates, it will break the test. If looking for a reset, there are a bunch of test automation vendors out there who come at the problem differently, using record and playback to accelerate authoring.

Some are using AI or multiple attributes to identify the selectors to address the stability issues. Our engineers believe in helping global teams get the best automation solutions for their projects and have created a scoring tool that i think that you will enjoy here. Cheers.

Answer 5

A useful tool to figure out what's failing and why is taking a screenshot on test failure and saving it alongside your test output. That way you can see what the page looks like at the exact moment of failure. This can be a huge help in figuring out why tests fail. A tutorial on how to take screenshots on failures can be found at Page object models.

Another thing that can help in some cases is to configure your driver to write a log file. In Java, you do this like this:

File logFile = new File(Paths.get(ctx.getOutputDirectory()).toString(), "chromedriver.log");
logFile.getParentFile().mkdirs();
logFile.createNewFile();
ChromeDriverService driverService = new Builder().withVerbose(verbose).withLogFile(logFile).build();
webDriver = new ChromeDriver(driverService, options);

This is far more technical but can help you find things like long loading times.

Answer 6

Tests are written by engineers. So to break the myth, it's the test architecture, test configuration, test environment, test structure, test code, etc are a few of the factors which in case of incompatibility makes our tests unstable.