How do I run parallel selenium scraper in grid?
I wanted to crawl 10 urls , and if I start two Firefox nodes, how do I request the nodes to crawl 5 urls in parallel?
I have gone through google, but found no help in this matter.
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It only takes a minute to sign up.Sign up to join this community
I created a project to do this and it reuses webdriver instances for better performance:
If configured correctly the grid can run 10 runs in parallel. The grid will manage which nodes run what.
Now you just need to start 10 test-runs and point them to the grid. E.g. starting a test run with a different URL.
You could create a script that loops and executes python with different parameters.
Okay, first of all you need to set up your grid to start with 5 Firefox nodes, you can do that by adding
-browser "browserName=firefox, maxInstances=5" –maxSessions 5 when you call the servers jar file. Then you need to parallelize your execution. The easiest way to do so is to pull the the url out of sys.argv and just start 10 processes and supply the url to crawl via the command line like
python3 yourcrawler.py https://your.url the https://your.url part will be stored in sys.argv inside the process and can be accessed from there. This will directly send 10 requests to the Hub, 5 of those will be executed immediatly, the other 5 will be queued until another node is free by the hub automatically (it's really a marvellous piece of software). If you need to collect the results and do something with them in the same process you can multithread your application, just create a class that inherits from threading.Thread and m,ove your crawler logic into that class, then you can spawn new threads by calling the class and instanciating it by supplying the url and let it do it's thing.