Python selenium grid parallel execution

Question

How do I run parallel selenium scraper in grid?

I wanted to crawl 10 urls , and if I start two Firefox nodes, how do I request the nodes to crawl 5 urls in parallel?

I have gone through google, but found no help in this matter.

Bhavani · Accepted Answer · 2018-02-22 17:09:23Z

0

You can use pytest for handling parallel execution.

Installation Steps:

pip install pytest pip install pytest-xdist pip install pytest-rerunfailures

After installation, you can execute with the command: Example: py.test -n 2 --rerun 2 test_sampletest.py

This command runs 2 tests in parallel.

answered Feb 22 '18 at 17:09

Bhavani

6231 gold badge3 silver badges17 bronze badges

I do not require the same test in parallel. I need to achieve this "I wanted to crawl 10 urls , and if I start two Firefox nodes, how do I request the nodes to crawl 5 urls each in parallel?" – smanjil Feb 23 '18 at 3:39
If you are not confined to python, please refer this solution mentioned in stack overflow with curl. stackoverflow.com/questions/42898289/… – Bhavani Feb 23 '18 at 7:05

add a comment |

chrismead · Accepted Answer · 2018-06-25 14:55:46Z

0

I created a project to do this and it reuses webdriver instances for better performance:

answered Jun 25 '18 at 14:55

chrismead

1382 bronze badges

add a comment |

Niels van Reijmersdal · Accepted Answer · 2019-04-21 18:34:42Z

If configured correctly the grid can run 10 runs in parallel. The grid will manage which nodes run what.

Now you just need to start 10 test-runs and point them to the grid. E.g. starting a test run with a different URL.

You could create a script that loops and executes python with different parameters.

3 Answers 3