How do I run parallel selenium scraper in grid?
I wanted to crawl 10 urls , and if I start two Firefox nodes, how do I request the nodes to crawl 5 urls in parallel?
I have gone through google, but found no help in this matter.
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It only takes a minute to sign up.Sign up to join this community
I created a project to do this and it reuses webdriver instances for better performance:
If configured correctly the grid can run 10 runs in parallel. The grid will manage which nodes run what.
Now you just need to start 10 test-runs and point them to the grid. E.g. starting a test run with a different URL.
You could create a script that loops and executes python with different parameters.