Selenium Webdriver get() shows intermittent TimeoutExceptions - how do I proceed?

Question

I'm using Selenium Webdriver for testing a Django/Python based web application. I use both, the phantomjs and firefox drivers. We use django's LiveServerTest test case, which basically starts up a test webserver for the app.

Now, on our CI environment (wercker), I sometimes - but not always - get a TimeoutException from my webdriver.get(url) calls. The thing is, though, that when I look at the screenshots (which I collect on exceptions) the actual page seems to be rendered just fine.

I'm already using a technique to wait for a page reload that is described here: http://www.obeythetestinggoat.com/how-to-get-selenium-to-wait-for-page-load-after-a-click.html. This wrapper basically waits until the old page's HTML element is replaced with the new one. It seemed to resolve similar problems for when I run my tests locally.

So, what am I experiencing here? How can I proceed? Any hints greatly apreciated.

Edit: I followed Artem's advice and remove all external resources (fonts and google maps calls). I initially thought that helped but then got other errors instead. I now get TimeoutExceptions in a click() call... Here's the log:

Traceback (most recent call last):
  File "/pipeline/build/flatfox/tests/ui/utils.py", line 57, in wrapper
    func(self, *args, **kwargs)
  File "/pipeline/build/flatfox/tests/ui/test_auction_cycle.py", line 157, in test_manage_applications
    self.driver.find_element_by_css_selector('.modal-dialog .btn[name=reject]').click()
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 65, in click
    self._execute(Command.CLICK_ELEMENT)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 385, in _execute
    return self._parent.execute(command, params)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 173, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 166, in check_response
    raise exception_class(message, screen, stacktrace)
TimeoutException: Message: u'{"errorMessage":"Click failed: timeout","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"81","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:55196","User-Agent":"Python-urllib/2.7"},"httpVersion":"1.1","method":"POST","post":"{\\"sessionId\\": \\"d67c58a0-610d-11e4-a75e-eb735e81fd4c\\", \\"id\\": \\":wdc:1414767284194\\"}","url":"/click","urlParsed":{"anchor":"","query":"","file":"click","directory":"/","path":"/click","relative":"/click","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/click","queryKey":{},"chunks":["click"]},"urlOriginal":"/session/d67c58a0-610d-11e4-a75e-eb735e81fd4c/element/%3Awdc%3A1414767284194/click"}}' ; Screenshot: available via screen 

Edit: Here's the code making the click():

with wait_for_page_load(self.driver):
    self.driver.find_element_by_css_selector('.modal-dialog .btn[name=reject]').click()

with

def wait_for(condition_function):
    start_time = time.time()
    while time.time() < start_time + 10:
        time.sleep(0.2)
        if condition_function():
            return True
    raise Exception('Timeout waiting for {0}'.format(condition_function.__name__))

class wait_for_page_load(object):
    def __init__(self, browser):
        self.browser = browser

    def __enter__(self):
        self.old_page = self.browser.find_element_by_tag_name('html')

    def page_has_loaded(self):
        new_page = self.browser.find_element_by_tag_name('html')
        return new_page.id != self.old_page.id

    def __exit__(self, *_):
        try:
            wait_for(self.page_has_loaded)
        except Exception, e:
            logging.getLogger("test.ui").error("wait_for_page_load(): failed with %s", e)

I'm a bit lost now and wonder whether it has something to do with the CI environment...

Thanks, Bernhard

Answer 1

In selenium the wait command doesn't really wait sometimes. That is there is some problem with that command.

You can try the time.sleep(); command of python. It is an equivalent of the Thread.sleep(); command of Java.

It will pause the execution of code for defined time limit. So you can give sufficient time that will allow your page to load.

Hope it works... :)

Answer 2

wait = WebDriverWait(driver, 10) 
element = wait.until(EC.element_to_be_clickable((By.ID,'someid'))) 

Try above code and see if it is working or not.

Answer 3

driver = webdriver.Firefox() 
driver.implicitly_wait(10) # seconds 
driver.get("http:// enter your URL.") 
myDynamicElement = driver.find_element_by_id("myDynamicElement") 

Try above code and see if it is working or not.

Answer 4

The possible reason might be that page looks to be rendered (UI elements are in place), but not completely loaded in browser.

If there is any background calls, especially to 3rd-party services, it might take a while to reach them and get response, only then browser is able to mark the page as finally loaded.

Answer 5

thank you all for your suggestions. Though it's still not perfect, I now have a better working environment. I combined Artem's suggestion of removing external dependencies with Pushpendra's tip to boost up implicit_wait and page_load_timeout further up. They are now at 12 and 30, but I intend to go even higher. I've also increased the sleep times, as suggested by milinpatel17, to as high as 4 seconds around calls that don't reload the page, but modify the DOM in a way.

The core of the problem seems to be that the CI environment's cpu loading sometimes is unpredictable and I have to make enough room for those times it is laggy. This is also what wercker's support suggested (nice guys, btw). I will now make these timeout values dependant on the environment, such that I can run local tests fast and then slow down on the CI.

I can't mark a single answer as valid, and, unfortunately, my reputation doesn't allow upvoting... sorry. But many thanks for your support!

Answer 6

Here is the ultimate solution specifically when you are dealing with Angular 7 or 8.

Instead of waiting for a longer duration using sleep or implicit wait methods, you can divide your wait time into the partition and use it recursively.

Below logic will wait for the page to render for a minimum of 300 seconds and a maximum of 900 seconds.

/**
 * This method will check page loading
 *
 */
public void waitForLoadingToComplete() {
    waitLoadingTime(3); // Enter the number of attempts you want to try
}

 private void waitLoadingTime(int i) {
        try {
  // wait for the loader to appear after particular action/click/navigation
            this.staticWait(300); 
  // check for the loader and wait till loader gets disappear
            waitForElementToBeNotPresent(By.cssSelector("Loader Element CSS"));                                                             
        } catch (org.openqa.selenium.TimeoutException e) {
            if (i != 0)
                waitLoadingTime(i - 1);
        }
    }

/**
 * This method is for the static wait
 *
 * @param millis
 */
public void staticWait(final long millis) {
    try {
        TimeUnit.MILLISECONDS.sleep(millis);
    } catch (final InterruptedException e) {
        System.err.println("Error in staticWait." + e);
    }
}

public void waitForElementToBeNotPresent(final By element) {
    long s = System.currentTimeMillis();
    new WebDriverWait(this.getDriver(), 30)
            .until(ExpectedConditions.not(ExpectedConditions.presenceOfAllElementsLocatedBy(element)));
    System.err.println("Waiting for Element to be not present completed. " + (System.currentTimeMillis() - s));
}