I'm not a QA guy, so I apologies if some of this may sound dumb or far-fetched.
Suppose we have a simple Selenium script where the tool queries a website's search field and records the response time.
Would it be possible to have 100*n virtual machines replicate this same action simultaneously, and record their respective response times? Our end-goal is to determine at what user count the response time begins to slow significantly.
In theory it would be possible yes, but most testers would use a load-testing tool. JMeter also supports Selenium based steps.
Selenium is a bit heavy for load testing. Most search fields could be tested with just http calls.
Theoretically yes, for example there is Selenium Grid project, as per description:
Its aim is to provide an easy way to run tests in parallel on multiple machines.
However it might be not the best idea because Selenium is browser automation framework and modern browsers are very resource intensive so it might be a better idea to convert your Selenium tests into a HTTP protocol-based performance tests.
The majority of load testing tools like Gatling or JMeter provide a HTTP Proxy server and given you run your Selenium tests via this proxy server you should be able to quickly get the "skeleton" of the performance test and after some parameterization/correlation you will be able to replay it with increased number of threads (virtual users).
Load Testing tools provide much more comprehensive metrics in terms of correlating the increasing load with the response time, transactions per second, errors per second, etc. moreover you will be able to simulate thousands of virtual users from a machine which only capable of launching several instances of the real web browser.
See How to Convert Selenium Scripts into the JMX Converter article for more details on the concept.
From personal experience, yes it's doable but it's not a comfortable solution. You can, in theory, connect infinitely many selenium node Docker containers to your hub (iirc the technical limitation is Javas LONG_MAX value of 9,223,372,036,854,775,807 but thats infinite for pretty much all intends and purposes) and deploy them to a cloud service like AWS. The selenium hub works reliable up to roughly 150 nodes per hub, upwards from there I found that it tends to drop sessions. So you'll need multiple Hubs as well and your test code needs to make sure none of them is used over capacity.
For the benefits, selenium triggers all the events a real user would also trigger including tracking, keep alives and so on but the price you pay is a lot of programming and devops work needed to implement an environment that can reliable create load on your page.
After all, my recommendation would be to use a specialized load testing tool like Apaches JMeter instead since it will probably give you more reliable (and reproducible) results with much less of a headache when implementing it.
Yes, the tool you want to use is Karate.
Here is a 10-thread example I made: https://github.com/djangofan/karate-selenium-grid-example/blob/master/src/test/java/features/google/form.feature
Then, you can install AeroKube Moon to get a free scalable 4-thread Grid cluster. If you want more threads, they are $5/thread/month. This is absolutely the cheapest solution.
Then, once you have that working, it is a simple matter to run the Karate test inside a Gatling test harness. It is about 20-40 lines of code to accomplish that. The reason you want to use Gatling, is because in advanced scenarios, you can run Gatling in a master-slave scenario using the Taurus test harness to control the slave machines which are all generating the load. (Taurus harness is the most advanced way to do it; its similar to using Blazemeter)
NOTE1: Make sure you don't run more load threads than you have available on the Moon cluster.
NOTE2: You can run Karate with a headless-chrome and then you don't really need the Moon Cluster.
Then, you have it.
