I am executing a script which clicks through a number of pages to retrieve data. After each click the new page obviously has to load and the time to load varies greatly (mostly 2-3 seconds, sometimes > 20 seconds). Thus, I need to make the code flexible to wait with executing until the element is loaded. I have implemented the solution below, but I still get a stale element reference error, which I do not understand (as the code should not exit the while loop before the loading is completed). Can anyone explain why this happens/ what I can do to resolve?
Sorry in advance, I am not able to share an executable script with you.
error = 1 if page_ref < Max_pagenum_1: while error == 1: try: link = browser.find_element_by_xpath('//li[contains(@class, "page-item") and contains(@class,"active")]/following- sibling::li/a').click() error = 0 time.sleep(3) except: error = 1 time.sleep(20) else: print('finished' + Name)
Max_pagenum_1 is the last page for each item page_ref is the currently active page