How do I run parallel selenium scraper in grid?
I wanted to crawl 10 urls , and if I start two Firefox nodes, how do I request the nodes to crawl 5 urls in parallel?
I have gone through google, but found no help in this matter.
You can use pytest for handling parallel execution.
Installation Steps:
pip install pytest pip install pytest-xdist pip install pytest-rerunfailures
After installation, you can execute with the command: Example: py.test -n 2 --rerun 2 test_sampletest.py
This command runs 2 tests in parallel.
-
I do not require the same test in parallel. I need to achieve this "I wanted to crawl 10 urls , and if I start two Firefox nodes, how do I request the nodes to crawl 5 urls each in parallel?" – smanjil Feb 23 '18 at 3:39
-
If you are not confined to python, please refer this solution mentioned in stack overflow with curl. stackoverflow.com/questions/42898289/… – Bhavani Feb 23 '18 at 7:05
I created a project to do this and it reuses webdriver instances for better performance:
If configured correctly the grid can run 10 runs in parallel. The grid will manage which nodes run what.
Now you just need to start 10 test-runs and point them to the grid. E.g. starting a test run with a different URL.
You could create a script that loops and executes python with different parameters.
answered Apr 21 '19 at 18:34
Okay, first of all you need to set up your grid to start with 5 Firefox nodes, you can do that by adding -browser "browserName=firefox, maxInstances=5" –maxSessions 5 when you call the servers jar file. Then you need to parallelize your execution. The easiest way to do so is to pull the the url out of sys.argv and just start 10 processes and supply the url to crawl via the command line like python3 yourcrawler.py https://your.url the https://your.url part will be stored in sys.argv[1] inside the process and can be accessed from there. This will directly send 10 requests to the Hub, 5 of those will be executed immediatly, the other 5 will be queued until another node is free by the hub automatically (it's really a marvellous piece of software). If you need to collect the results and do something with them in the same process you can multithread your application, just create a class that inherits from threading.Thread and m,ove your crawler logic into that class, then you can spawn new threads by calling the class and instanciating it by supplying the url and let it do it's thing.
