I am currently working on a project that involves the extraction of products name and price from an online store, using Robot Framework. The problem is that after I go over a few pages/links (around 200) the page redirects me to captcha. I use the sleep keyword to wait for 10 seconds before I go to the next page, but I still get captcha. What can I do better in order to trick captcha?

  • 1
    How does the administrator of the site feel about this project? You should be aware that buried in the terms and conditions of many sites includes a provision that you don't do automated scraping. – corsiKa Dec 13 '20 at 17:39

This is another captcha question.

If you're on a test environment, the best way for you would be to build in a way to work around it. This workaround would be of course turned on only on the test environment(s).

If you're scraping some production site, then you have no (easy) way to fill in the code to pass it. In your described situation, you might want to get rid of the captcha if you start a new session, e.i. you might want to switch into a new incognito mode after a certain number of visited pages. I don't know for sure if this will help, but it's worth the try.

Some other question here that might help you:

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.