I am currently working on a project that involves the extraction of products name and price from an online store, using Robot Framework. The problem is that after I go over a few pages/links (around 200) the page redirects me to captcha. I use the sleep keyword to wait for 10 seconds before I go to the next page, but I still get captcha. What can I do better in order to trick captcha?
This is another captcha question.
If you're on a test environment, the best way for you would be to build in a way to work around it. This workaround would be of course turned on only on the test environment(s).
If you're scraping some production site, then you have no (easy) way to fill in the code to pass it. In your described situation, you might want to get rid of the captcha if you start a new session, e.i. you might want to switch into a new incognito mode after a certain number of visited pages. I don't know for sure if this will help, but it's worth the try.
Some other question here that might help you: