I am executing a script which clicks through a number of pages to retrieve data. After each click the new page obviously has to load and the time to load varies greatly (mostly 2-3 seconds, sometimes > 20 seconds). Thus, I need to make the code flexible to wait with executing until the element is loaded. I have implemented the solution below, but I still get a stale element reference error, which I do not understand (as the code should not exit the while loop before the loading is completed). Can anyone explain why this happens/ what I can do to resolve?
Sorry in advance, I am not able to share an executable script with you.
error = 1
if page_ref < Max_pagenum_1:
while error == 1:
try:
link =
browser.find_element_by_xpath('//li[contains(@class,
"page-item") and
contains(@class,"active")]/following-
sibling::li/a').click()
error = 0
time.sleep(3)
except:
error = 1
time.sleep(20)
else: print('finished' + Name)
Max_pagenum_1 is the last page for each item page_ref is the currently active page