How do you calculate the number of virtual concurrent users to use in a load test given data on actual site traffic?

Question

When working on performance testing, one of the problems I've faced is trying to translate real-world web application traffic to simulated traffic.

A practical example is for an eCommerce site. During a user's experience on the eCommerce site, they will be adding and removing items from the shopping cart, logging in and/or editing their contact information, updating their shipping information and preferences, applying payments, and displaying receipts or, in some cases, product for download.

As an example, a client reports a total number of shopping carts created during an hour to be 1000 with a total over the course of a 24 hour period to be 10000. Not all shopping carts are taken through to completion but all shopping carts have gone through some part of the above steps before fulfillment or abandonment. Let's say "think time" per page is about 30 seconds.

This is all "off the top of my head" scenario with no correlation to a real-life situation, it's more trying to find a generalized "rule of thumb" for figuring out how many Virtual Users to use. In the past when I've made judgement calls, I've had a hard time justifying the numbers I've used. In the above example, my judgement call would be based upon throughput. How many shopping carts were completed to fulfillment within an hour? If that number is 400, then through trial and error, I can configure a number of virtual users that would approximate that number of carts fulfilled in the hour.

That's all still based on guess work with nothing really concrete in the reasoning behind it. So, how can you determine the number of concurrent virtual users to configure in your load test to test performance with the above load?

Answer 1

All right, So Tristann kindly revised his original question to include more details in terms of a scenario. So I'm adding a second answer to more directly address it.

Firstly you'd probably want to ask a few more questions about what the customer is most concerned with and what they want tested, here's a small sample:

• what's the duration of the shoppers that complete purchases look like for established users? min, average, max
• How many completed carts are first time shoppers and are creating an account during checkout. (and how much time does that add on average?)
• What's the average duration on the site for shoppers that abandon their carts
• how many items are in the average shopping cart that is fully processed.
• how many items are in the average abandoned shopping cart.

Answers to those will go towards figuring out how many different shopping scenarios you need, and how you go about tailoring think times, and what ranges you use when randomizing them. As you can start to see it looks like we'll want three scenarios at least, first time shopper, returning shopper, abandoned cart. And you'll have an idea how many items you need to search for and put in the cart for each scenario etc.

Lets make up a few answers and focus on a returning shopper scenario. within a 'peak hour' test. We'll presume that it's 10 minutes on average with a low of 5 minutes and a high of 15, and that 3/4 of the users were returning users.

So when you create your loadtest scenario, you'll want to time it with no think times, and then adjust reasonable think times at each step so that it takes about 10 minutes to complete the script with think time enabled. When the script is run, you'll want to set randomization on the think times to allow for +/- 50% of the stated times.

So, for a peak hour test, we'll want 600 iterations of our 'abandoned cart' scenario, 100 of the 'new user' scenario (that registers a new user during checkout) and 300 iterations of our returning user scenario.

So for the returning user script, if it takes 10 min average to run, a single vuser can run the script 6 times during our test. To get 300 iterations total we'll thus need to use 50 vusers in parallel. over that one hour (50*6=300)

If we presume for a moment abandoned carts is a 4 minute duration, then each vuser running that scenario can perform 15 iterations (60/4) during the hour. So to get a target of 600 iterations of the abandoned scrip to run in an hour, you would need (600/15) 40 vusers running that scenario during the loadtest.

And if the new user script takes 15 minutes average to complete, then you get 4 iterations per user in an hour and you'd need 25 vusers running that scenario to get 100 iterations over the course of an hour.

So your total ends up being 40+50+25 or 115 vusers to model the existing peak hour. If you want to simulate an even bigger burst of traffic within that hour, you could use even more vusers, and have the loadtest tool ramp them up and then back down so you still get your iterations but have a larger peak load in the middle of the test.

And (presuming you've created LOTS of test data already) if the customer wants to see if the site can sustain 4 times the current load, then you could run the same scenarios but using 160+200+100= 460 vusers total instead of 115

Answer 2

I wrote about concurrent users and numbers in a blog post: http://blog.xceptance.de/2011/06/07/get-the-right-load-mix-out-of-a-few-numbers/

Wait… where are my concurrent users? This is simple: “concurrent users” is an inaccurate way of describing traffic, so we have not used that number yet. Why is that?

To get to the bottom of that, we simply check how long a visit takes. Depending on the shop, an average visit might take 2 to 4 minutes. Successfully shopping might take 15 minutes. If we expect about 10 page views per visit and a page view takes 1 second to load and 20 seconds to read it (already a really really high number for an average), a visit would take 10 * 1 second + 9 * 20 seconds = 190 seconds.

Let’s go with the 190 seconds for a visit on average. If we just could serve one visitor at a time, we could serve 60 minutes (3600 seconds) / 190 seconds per visits = 19 visitors per hour. But because we would like to serve 10,000 per hour, we have to deal with 10,000 / 19 = 526 visitors at the same time. This is the famous concurrent user number.

If we now double the think time, we have 1,052 concurrent users/visitors. If we cut it down to 1 second think time, we will get a visit length of 19 seconds and therefore 10,000 visits / (3600 seconds / 19) = 53 concurrent visitors.

Answer 3

NOTE: this answer was to an earlier version of the question which was asking of you needed a 1:1 relationship between vusers, etc. Rather than re-word it I'll let it stand as is since the info inside is still pretty sound. But now you know why it doesn't seem to be directly answering the new version of the question.

Generally if you can afford it (e.g. licenses for vusers, hardware for your test rig that's running the vusers) the best answer is to use a 1:1 relationship. This means including think time between all the steps in your load scenarios, and can require a substantial number of vusers. Now with some systems like open source, or VSTS Ultimate (which now has an unlimited vuser license) the cost to license vusers is not that great, and since well designed systems can run upwards of a thousand vusers from a single cpu, the hardware cost to do this is not that great.

The reason this is best has many factors but the most important ones are

• that it forces the server side to maintain and update parallel sessions for each of the vusers, so the stress on the servers is about as realistic as you can make it.
• If your transaction intervals and think times are randomized, you still have the potential to come close to the maximum 'rush' or 'burst' of users in a given small timeframe

The latter point is very important both if we are talking about simul users relative to a very short timeframe such as 1-10 seconds, or relative to a larger duration such as users per hour. If you are compressing think times and using 100 vusers repeating a script that takes 36 seconds to simulate the load of 10000 users over an hour's time, the maximum 'peak' you can generate during that period is 100 requests. If you use 1000 vusers each repeating a script that takes 6 minutes, your maximum peak is 1000 requests, much closer to what could realistically happen.

Note however that in this scenario, I'm NOT using 10,000 vusers to represent the load on the site over that hours time. Since the test is one hour long (probably representing the peak hour in the day) and the script only takes on average 6 minutes (including think time) to execute, I can re-use that vuser to simulate the load of up to 10 normal users over the hour duration. OTOH if the peak load was 2000 users 'at once' during that hour, what I'd want to do is use 2000 vusers, and have them wait an average of 6 minutes between script iterations, so that each executes about 5 iterations of the script over the hours time.

OTOH if you are just trying to model 'background load' of non-logged in guests who are browsing static pages, then condensing think times and using a single vuser to simulate the transactions from multiple 'simultaneous' users over a period of time is still fairly realistic, and can save you a bit of money when your cost per vuser is high. Just be aware that this has the effect of creating a very smooth continuous load, which is not what happens in real life.

BTW: the MS Patterns and Practices folks created an awsome free book (I procured a printed copy I liked it that much) on performance testing that is pretty much platform agnostic and very very well worth reading for anyone doing performance or loadtesting. You can get it from this Codeplex Link

Answer 4

The way that has worked for myself, and that I was taught by others, was to scale down the total number of concurrent users and think time (time where a user is not doing anything but reading), and then decrease the gap between when one user leaves and another one arrives. It takes some math, and a lot of trouble-shooting, but, it has worked for some basic performance testing.

Answer 5

I have created an iOS app from which you can set/calculate all you performance testing scenarios on your iPhone/iPad/iPod touch.

Hope it helps the performance testing community.

App Store Link.

Support Site/Tutorial Link.

If you need help shoot me an email at my support site.