3

The AUT is built with LAMP* Stack and is the Intranet Home app and is going to be used by almost all employees of a large corporation. Application Performance Testing is carried out using HPE suite of tools (LoadRunner). The average user load is determined to be 150 users, average hourly page views is determined to be approx 4000/hour and we already did an endurance test as described below:

  • Ramp up 75 users, 2 every 20 seconds
  • Run for 6 hours
  • Ramp down 30 users from it simultaneously
  • Run the 45 users for another 4 hours
  • Ramp down another 15 users simultaneously
  • Run for another 2 hours with the users(30) left
  • Then ramp up another 45 users and repeat above steps for next 12 hours.

I have two questions:

  1. What is wrong with the above strategy? Should we have run the test for a lower transaction target but using the same number as baseline (150) users?
  2. What should be the key metrics we should be looking for as a part of the endurance test? Where should we be looking for answers? I am not talking about 'Long Term Application stability' kind of answers. More details would definitely help

*LAMP Stack - Drupal, Varnish & Redis for Caching, Apache, MySQL/MariaDB mostly on Oracle Linux.

Let me know if you would like more details. Thanks for all the help so far.

Improve this question

2 Answers 2

Reset to default
3

Given 4000 requests per hour and the fact the application will be used only internally (response time is not crucial) I don't think you have to do any load testing as 4000 requests per hour is just 1.11 requests per second.

Normally performance testing is done as follows:

  • Load Testing - put your application under anticipated load (in your case 150 concurrent users) for short period of time (1 - 2 hours) to see what response time looks like, whether it meets your expectations, are there any errors, etc. Make sure to increase and decrease the load gradually to see the trends (i.e. what is the correlation between virtual users number increased twice and application response time, throughput, etc.)
  • Soak Testing - slightly less virtual users (i.e. 100) but for longer period (overnight or weekend) to see if there are memory leaks or other unexpected errors.

  • Stress Testing - basically checking application boundaries, i.e.:

    • how many concurrent users can be served while response time is acceptable (doesn't exceed SLA)
    • what is the maximum amount of concurrent users which may be served unless errors start occuring
    • how does the system behave when the load gets back to normal (does it recover)
    • what is the first bottleneck, i.e. which component fails first (NB. it may be lack of resources as well so make sure you monitor baseline health metrics on application under test side)

See Why ‘Normal’ Load Testing Isn’t Enough article for more information on above performance testing techniques and why they all are required.

Improve this answer
2

In my personal opinions, I have a few points (in strong font.) to add:

  • The average user load is determined to be 150 users, what is your peak user load?
  • average hourly page views is determined to be approx 4000/hour, what is your peak page view per hour?
  • Ramp up 75 users, 2 every 20 seconds and run for 6 hours, by the end of 6 hours, would you have 75 users in total? It is not really clear why you would not have 150 users (the average user load)
  • Ramp down 30 users from it simultaneously, I am assuming this 30 users have logged off and gone home, but where you got this number 30 from? It that a guess?
  • Run the 45 users for another 4 hours, with 30 users gone, now you have 45 users left.
  • Ramp down another 15 users simultaneously, is 15 a guess or an average value?
  • Run for another 2 hours with the users(30) left, most users are offline, only 30 left.
  • Then ramp up another 45 users and repeat above steps for next 12 hours.

You have done a very good job on simulating how the amount of load would vary over a course of a day, well done. Below are my suggestions:

  • You mentioned the average load is 150 users, yet you were testing against 75 users; in reality, the load would be heavier than you have tested.
  • How you calculated the number of users is unclear as I stated above.
  • You have not explicitly tested against hourly page view (4000) yet.
  • Test approach is linear in nature and has sudden chop-offs. E.g. you add 2 users every 20 seconds and 30 users & 15 users are removed out of sudden. In reality, it is unlikely for users to log on linearly and log off suddenly all together. May I suggest you introduce a more realistic logging on and logging off trends?

Key matrices that may be useful to you:

  • Average response time, it may become degraded over time
  • Average time between failure, how long does it take for your system to fail
  • Average time for a user to stay, e.g. between log in and log off
Improve this answer
3
  • Thank you, your questions bring a lot of clarity to the thought process in my head. As I already mentioned, this test was already done. This was a badly planned test devised without asking the kind of questions you just did. Those kind of questions are what I am trying to find out as well.
    – petwolfe
    Commented Jan 23, 2017 at 23:35
  • To Answer your questions: We do not have the peak load numbers. Also, the launch is expected to be heavy and no one has an idea as to how big it is going to be. Same with page views. Yes, it was decided to go with 75 users instead of 150 users. Most of these were random decisions taken based on guesswork. The scripts and test scenarios are designed in a way as to achieve the 4000 page views / hour target. We are able to hit it almost always.
    – petwolfe
    Commented Jan 23, 2017 at 23:37
  • @petwolfe, my pleasure. Come back any time should you need to have another discussion.
    – Yu Zhang
    Commented Jan 23, 2017 at 23:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.