14

I am running a few scripts (in my personal and business laptop) and every day I get a different error at a different step.

Sometimes I run and it is passed, but I run again and get an error that an element is not visible or so.

I've included implicit waits, explicit waits, also time for page to load, however it doesn't help. How can I fix this?

4
  • 6
    I work in Testim.io, we did research on this - how in depth do you want the answer? A lot of it is using sleeps and using CSS selectors to select things. Another big part is how actual clicking work and that being different across OSs and browsers. We have like... 100K LoC normalizing just how to click... – Benjamin Gruenbaum Oct 29 '20 at 13:33
  • 2
    We also wrote a FOSS tool to help with this but it's only available for puppeteer/playwright - let us know if you want to see it for Selenium github.com/testimio/root-cause – Benjamin Gruenbaum Oct 29 '20 at 13:34
  • @BenjaminGruenbaum If you're willing to write a blog-post-size answer about it, I don't think that's a problem. Just make sure you're disclosing affiliation in your post when referencing products your company/organization has worked on. – Mast Oct 30 '20 at 12:13
  • 3
    @Mast it might sound ironic, but because I am affiliated to an entity/company working on this - I try to avoid writing about this (I'd rather answer things in StackOverflow or work on OS stuff in GitHub) - I have given talks about this in several places like JSConf (with affiliation disclosed, <3 Japan) - here are some slides: docs.google.com/presentation/d/… (Testim is mentioned in the logo but the slides don't mention it otherwise) – Benjamin Gruenbaum Oct 31 '20 at 11:57
27

Main observations in unstable scripts are as follows:

  1. Selenium tests are usually unstable because of the unnecessary complexity created in your test framework
  2. Enabling parallel execution without keeping in mind that one test may affect the state of the system that in turn affect the other test running in parallel.
  3. Using deprecated modules
  4. Incorrect configuration of modules that are used in the framework
  5. Not using explicit wait for asynchronous actions
  6. Using tools that are not meant for testing the software under test (e.g., use protractor for angular webpages than selenium)
  7. Developing test in one OS and running it in another OS, in CI/CD test agents may have different OS and configurations than what is used in the development PC

There are many such reasons. Also, whenever investigating such issue, follow the below steps in the order mentioned:

1. Trust your code and doubt software under test (SUT):

If everything was working fine and the test starts to fail suddenly. Instead of debugging your code for issues, start with checking the actual product. Do some visual inspection and see whether the development team has modified the element or the element is no longer being displayed.

2. Trust your code and doubt the environment:

If everything was working fine locally and failed as soon as you integrated to CI/CD. Then investigate the product behaviour in the test server. Mostly due to OS and configuration differences the product won't work as it would in local (raise a bug)

3. Now doubt your scripts (using absolute XPATH):

You might be using an absolute XPATH. This causes flaky tests when the DOM structure changes. Use a relative XPATH (CSS would be more recommended). Never use xpath/ if you have a unique ID/name to identify an element.

4. Now doubt your scripts (not using explicit wait):

Sometimes scripts lack explicit wait and try to interact with dynamic elements, and this causes the test to fail, because it tries to interact with the element before it is even available in the DOM.

5. Now doubt your scripts (handling spinners):

Sometimes spinner takes time to get displayed. So, if you are just checking for the invisibility condition then it will return true and try to interact with the next element before the actual spinner event completes.

So, first, check the visibility of spinner and then check for invisibility before interacting with other dynamic elements.

6. Now doubt your scripts (not handling iFrames):

Sometimes an element will be inside iframes and scripts won't switch between frames before interacting with these elements.

Check whether any parent element contains the tag frame or iframe to determine whether the element is inside an iframe.

12
  • 5
    I was going to say "your tests are unstable because they are poorly written", but this answer is so much better. – Mate Mrše Oct 29 '20 at 7:56
  • 1
    Great answer. As an extension to point 3, I'd also say - if you don't have unique ids/names then ask for them! It's often trivial for developers to add this in but the benefit to you for automation is huge. – Moorpheus Oct 29 '20 at 12:31
  • 1
    @PDHide the point I was trying to make (I agree with your answer which is why I upvoted it - and was replying to the first comment and not your post) is that even if it sounds easy to write "non-poorly written tests", once you have a sufficient number of them even a small percent breaking can be terrible. – Benjamin Gruenbaum Oct 29 '20 at 15:46
  • 1
    Hopefully, if the you're even remotely Agile and developers have made an intentional change to something with automated tests, you'll be kept "in the loop" and know to expect the failure in advance, in which case you can skip some of these steps. Though if your development team still follows Waterfall (aka "Keep the designs secret from the developers and the development secret from the testers"), this may not be the case. – Andrew Ray Oct 29 '20 at 18:39
  • 2
    @PDHide I mostly agree - but I think making tests stable once you get to rather large scale - can get very tricky. I've seen a lot of E2E tests by big companies (Microsoft/Salesforce/Autodesk/Wix etc) and some breakage is surprising (Testim noramalizes all of this..). Some examples: updating the React/Angular version, updating libraries, new Chrome version, new ChromeDriver/SafariDriver/GeckoDriver version, grid xvfb version updates, OS differences within linuxes and x11 that cause input_injector_x11 behave slightly differently, a/b testing tools and SDKs and of course - new developers :] – Benjamin Gruenbaum Oct 29 '20 at 19:44
6

The more tests are integrated, the less stable they are. This is because they involve many more environmental factors.

Basically, Selenium tests are not just integrated in terms of software components which compose your AUT, but also in terms of the particular version of the OS, particular browser version, particular graphical driver version, or "headless" display implementation...

They are also "integrated" with numerous libraries which all have their own defects and other specific, with particular networking infrastructure which might cause routing or messaging issues, with your hardware that might not be just powerful enough for rendering the components in a web browser in stable and reasonable time.

So to make your tests stable, you have to stabilize all those factors. If this issue is really critical for you, it is probably better to organize your test execution in virtualized environments where you would be able to control them.

4
  • 1
    Plus introduce shift-left testing, i. e., have integeation tests (e.g. at REST API level) than end-to-end tests. That would help to isolate tested components a bit. – dzieciou Oct 28 '20 at 18:44
  • What is "AUT"? – Peter Mortensen Oct 31 '20 at 2:29
  • Application under test – Alexey R. Oct 31 '20 at 8:38
  • One linguistic comment regarding the first sentence: these are not the tests that are being integrated but rather tested components. Perhaps, it should be: The more components the integration test tests, the lest stable it is. – dzieciou Jan 6 at 10:33
2

You may be running synchronous tasks against an asynchronous device - a web browser.

Modern browsers, frameworks and pages frequently make calls, such as to API's, vendors, etc. that are done asynchronously. You tests may not account for this behavior and if not will consequently fail intermittently because of it, combined with other factors such as detailed by Alexey.

1

Great question! Creating and debugging a stable Selenium test can take 1–2 days. In addition to the excellent responses in this thread, if you are using Selenium today, then you no doubt run into errors like these every time there is the slightest code change:

-NoSuchElementException -ElementNotVisibleException -TimeoutException -NoSuchAttributeException

This is because Selenium uses single selectors to identify elements and if they change because of feature or code updates, it will break the test. If looking for a reset, there are a bunch of test automation vendors out there who come at the problem differently, using record and playback to accelerate authoring.

Some are using AI or multiple attributes to identify the selectors to address the stability issues. Our engineers believe in helping global teams get the best automation solutions for their projects and have created a scoring tool that i think that you will enjoy here. Cheers.

1

A useful tool to figure out what's failing and why is taking a screenshot on test failure and saving it alongside your test output. That way you can see what the page looks like at the exact moment of failure. This can be a huge help in figuring out why tests fail. A tutorial on how to take screenshots on failures can be found at Page object models.

Another thing that can help in some cases is to configure your driver to write a log file. In Java, you do this like this:

File logFile = new File(Paths.get(ctx.getOutputDirectory()).toString(), "chromedriver.log");
logFile.getParentFile().mkdirs();
logFile.createNewFile();
ChromeDriverService driverService = new Builder().withVerbose(verbose).withLogFile(logFile).build();
webDriver = new ChromeDriver(driverService, options);

This is far more technical but can help you find things like long loading times.

0

Tests are written by engineers. So to break the myth, it's the test architecture, test configuration, test environment, test structure, test code, etc are a few of the factors which in case of incompatibility makes our tests unstable.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.