Calculate Concurrent and Simultaneous users

Question

We will have to benchmark our application by finding maximum number of concurrent and simultaneous users supporting. We have been using JMETER

We did this,

1. Created concurrent Thread group
2. Specified the concurrent user count as 100
3. Specified the Hold time as 5 mins
4. Executed the thread

The execution triggers around 20k requests totally and kept 100 active threads at any given point of time. All the requests are successful and response time is within SLA. Beyond 100 the response time doesn't meet the SLA.

Q1: Are we doing correct? Is this correct way to find concurrent users count?

Q2: We are doing the same test for simultaneous users, Only difference is that we execute multiple thread group simultaneously with think time. Is it right way to calculate simultaneous users?

Answer 1

I would suggest the following approach:

1. Make sure that JMeter virtual users represent real users as close as possible, to wit:

   • HTTP Request samplers are configured to download embedded resources and use concurrent Thread Pool to download them - this will allow to fetch images, styles and scripts like browsers do. You can do it using HTTP Request Defaults
   • Use HTTP Cache Manager to represent the browser cache as real browsers download these images, scripts and styles only once, on subsequent requests they are being returned from disk cache
   • Use HTTP Cookie Manager to handle user sessions and deal with cookie-based authentication
   • Use Timers to represent "think time" as real users don't hammer the server non-stop, they need some time to "think" between operations

2. Once done you can conduct so called load testing - put the system under anticipated load, i.e. simulate as many users as you expect to see whether your system is capable of handling them providing reasonable response times and without errors.

3. If point 2 is successful - start gradually increasing the load until response time starts exceeding acceptable upper boundary or errors start occurring. Mind correlation between increasing load and increasing response time. This way you will know what is the maximum amount of users your system can support. Once you hit the bottleneck gradually decrease the load to see whether system recovers and starts operating normally or not. At this point it would be also good to identify the cause of increased response time, like high CPU usage or slow disk or slow DB queries, etc. so you could report the root cause of the performance bottleneck. This approach is called stress testing
4. It would be also good to consider soak testing: putting the system under anticipated load (or a little bit more) for a longer period of time - this way you will be able to see how does it behave under a sustained load and detect any memory leaks

See Why ‘Normal’ Load Testing Isn’t Enough article for more detailed explanation.

Remember one point: there should be always 1-1 mapping of JMeter virtual user to system real user, for example re-using one login or running test without think times doesn't tell the full story, your test needs to be realistic.