Single User test shows latency while load test passes for web application. Where to look for clues?

Question

I am testing a Drupal/CMS-like Web Application for performance issues and have come across a strange situation. The application came for testing after users complained about perceived slowness. After initial analysis it was decided to do a load test for ~30 users (using HPE Load Runner). Steps performed are as below:

1. Post scripting, as a preliminary test, we ran an NV Analysis¹ for a single user and found out that a lot of the transactions had spent time waiting at the server. Application team was duly notified.
2. Next, we went into application load test. Surprisingly the load test came clean, with all the transactions well within stipulated time range.
3. Next, we added some background noise to the application by hitting random URLs in the application while the load test is run to generate some 'noise'. Again, as you guessed, the results were fine.
4. Next, we created more noise by generating hits across other applications deployed in the test environment which shares the infrastructure with our AUT in production as well. This, to our dismay, gave better response times than previous tests.

I am at a loss right now. Where should I look for solutions right now? Of course, the NV report is being looked at as of now. Also, another short load test with all virtual users being pushed into the test at once (instead of a slow ramp up) is is being planned.

¹_{This is similar to yslow where a rule based diagnostic report is provided for each page under test. More details here.}

Answer 1

There are a number of possibilities:

• It may be that the load test is reusing existing connections instead of closing and reopening connections with each request. This would cause the load test to run better than the single user test.

• Make sure caching is turned off for the load test. Caching at the DB & web servers as well as the load tool must be considered.

• Are the single-user test and the load tool using the same hardware and traversing the same physical paths to the servers?

• Is time of day a factor? Is there something else that may be running concurrently to slow down the single user test? Watching the % CPU usage counter, the MB RAM Free counter, the Network Bytes/sec in & out counters, and the % Disk Usage counters may be helpful here. These counters can tell you if the source or target machines are being overloaded in some way.