How do I run parallel selenium scraper in grid?
I wanted to crawl 10 urls , and if I start two Firefox nodes, how do I request the nodes to crawl 5 urls in parallel?
I have gone through google, but found no help in this matter.
You can use pytest for handling parallel execution.
pip install pytest pip install pytest-xdist pip install pytest-rerunfailures
After installation, you can execute with the command: Example: py.test -n 2 --rerun 2 test_sampletest.py
This command runs 2 tests in parallel.
I created a project to do this and it reuses webdriver instances for better performance:
If configured correctly the grid can run 10 runs in parallel. The grid will manage which nodes run what.
Now you just need to start 10 test-runs and point them to the grid. E.g. starting a test run with a different URL.
You could create a script that loops and executes python with different parameters.