1

How do I run parallel selenium scraper in grid?

I wanted to crawl 10 urls , and if I start two Firefox nodes, how do I request the nodes to crawl 5 urls in parallel?

I have gone through google, but found no help in this matter.

0

You can use pytest for handling parallel execution.

Installation Steps:

pip install pytest pip install pytest-xdist pip install pytest-rerunfailures

After installation, you can execute with the command: Example: py.test -n 2 --rerun 2 test_sampletest.py

This command runs 2 tests in parallel.

  • I do not require the same test in parallel. I need to achieve this "I wanted to crawl 10 urls , and if I start two Firefox nodes, how do I request the nodes to crawl 5 urls each in parallel?" – smanjil Feb 23 '18 at 3:39
  • If you are not confined to python, please refer this solution mentioned in stack overflow with curl. stackoverflow.com/questions/42898289/… – Bhavani Feb 23 '18 at 7:05
0

I created a project to do this and it reuses webdriver instances for better performance:

https://github.com/testlabauto/local_selenium_pool

https://pypi.org/project/local-selenium-pool/

0

If configured correctly the grid can run 10 runs in parallel. The grid will manage which nodes run what.

Now you just need to start 10 test-runs and point them to the grid. E.g. starting a test run with a different URL.

You could create a script that loops and executes python with different parameters.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.