I'm trying to scrape information from an array of links using selenium. Each of them take ~2 seconds to scrape, and it adds up to quite a while.

Is it possible / worth trying to open the links in separate tabs (or sets of tabs?) in order to multitask scraping information? Is there an otherwise better method to optimize iterating through these pages?

I'm using a headless Chrome driver and Python.

  • Selenium is slow. Is there any reasons you are not scraping on HTTP level? E.g. With scrapy. – dzieciou Feb 7 '20 at 5:23
  • 1
    There is some JS on the site that I have to wait to load – Spiralio Feb 7 '20 at 5:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.