python selenium - waiting for item to load (stale element reference error)

Question

I am executing a script which clicks through a number of pages to retrieve data. After each click the new page obviously has to load and the time to load varies greatly (mostly 2-3 seconds, sometimes > 20 seconds). Thus, I need to make the code flexible to wait with executing until the element is loaded. I have implemented the solution below, but I still get a stale element reference error, which I do not understand (as the code should not exit the while loop before the loading is completed). Can anyone explain why this happens/ what I can do to resolve?

Sorry in advance, I am not able to share an executable script with you.

error = 1 
        if page_ref < Max_pagenum_1: 
            while error == 1: 
                try: 
                  link = 
                    browser.find_element_by_xpath('//li[contains(@class, 
                    "page-item") and 
                    contains(@class,"active")]/following- 
                    sibling::li/a').click()
                    error = 0
                    time.sleep(3)
                except: 
                   error = 1
                    time.sleep(20)
        else: print('finished' + Name) 

Max_pagenum_1 is the last page for each item page_ref is the currently active page

Answer 1

You should wait when xpath is visible before clicking like that

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

WebDriverWait(driver, 15).until(EC.visibility_of_element_located((By.XPATH, '//li[contains(@class,"page-item") and contains(@class,"active")]/following-sibling::li/a'))

It means, wait until xpath is visible (max 15 second), then you can continue to click.

Answer 2

This is a classic problem with using Xpath and then calling a click method instantly.

before running the click method on it you need to find the element, then pause , check its still active, then run the click method on it.

you should also avoid having a catch all exception handler, instead specify the exceptions you want to catch. If you aren't specific you program will be confusing when it goes wrong as it conflate all errors.