We want to load test a web application using load/stress tests in Visual Studio Enterprise 2015. With the number of concurrent virtual users in mind, how do we know when running these tests locally is not enough (as in when can one machine not provide sufficient load for the tests)? Similarly, if we move to a test controller and a small number of agents on-premises, how can we figure out the limits of our test setup?
The data is sensitive, so we'd rather not move to Visual Studio Online.
The number of virtual users (VUs) that one computer can handle depends on the complexity and duration of the individual test cases. Many factors contribute to the complexity, including: number of requests, number of plugins called and what they do, number of extraction and validation rules. As complexity increases so does the usage of memory (RAM) and CPU time.
My rule of thumb is to assume that one computer can support between 250 and 500 VUs. Having done some basic analysis I have a better understanding of how the tests operate and how many computers will be needed. Microsoft publish these recommendations for the VUs per computer.
If your tests (or some of them) are available try running a load test on one computer with 50 VUs and repeat with 100 or 200 VUs. Look particularly at memory and CPU usage. That should give some data from which you can estimate what one computer will support, but remember to leave plenty of headroom in both measures. (I do not want to commit myself on how much headroom to leave.)
Another thing to consider is the network bandwidth available between the test computers and the system under test. The bandwidth on some network links may limit the number of VUs that can run your tests.
(There was a small confusion in the first version of this question. When spreading the VUs across multiple computers, you will need one controller and several agents. The controller manages the test and collates the results into the results database. The agents computers execute the tests.)
