Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

When running automated performance tests, is it better to preload any DLL's which would slow the first run of a test? This would improve the consistency of the test results but would not reflect the worst case scenario.

How do you handle this issue in your performance tests?

share|improve this question

3 Answers 3

up vote 3 down vote accepted

I'm of two minds about this.

  1. Don't load them - There are a number of problems I've uncovered with a web application where the DLL's and other such objects were not loaded in memory first. Two concurrent users hitting the app at the same time before all files were "spun up" created some really strange concurrency issues that only happened in that instance. If the purpose of your testing is to check these initialization conditions under load, then definitely don't load them DLL's first.
  2. Load them up - For many applications, the application has been up and running for a while the next time someone hits it so most general performance testing is most "realistic" when the DLL's are pre-loaded.

Essentially, two different kinds of testing require two different answers.

share|improve this answer
    
The kind of testing I'm interesting in is measuring performance. Thus, these tests are run one at a time. So I think you're second argument applies to my case. –  Alex B May 19 '11 at 12:52
1  
"First run" performance might be something you want to check. So, these can both be performance tests. I'd say don't load for most performance tests, for consistency; but be aware that you are missing some key scenarios, and decide if you need to make first run performance tests. Keep in mind that first run is your end user's first impression. I've also found that performance testing is often where the most interesting concurrency issues are discovered on large teams. –  Ethel Evans May 19 '11 at 17:16

If you look at any of the standard benchmarks available from TPC, SPEC, etc.. you will notice that all of them include a warm up period in the performance test, a time under full load, and then a ramp down interval. This ramp up period is designed to allow code which needed to be compiled upon first execution or libraries to be loaded to take place before the time under full load interval is entered into in order to measure optimal performance.

This ramp up/warm up period is established best practice in the commercial benchmark space, there are an awful lot of flat out poor/untrained performance testers in the market who simply go from 0 to 100% of load with few steps to allow for a warm up interval of any type. There is a market analogy for this called the "tea totaller." It goes something like this, "Take a Tea Totaller and a bottle of Vodka. Combine rapidly. You get total system shock. On the other hand, take a tea totaller, a bottle of Vodka and a thimble full of vodka at a time and you are likely to have a smoother ramp up to your test and you are also likely to find the exact spot at which your system melts down as load is increased....which is often at a point greater than the combine all at once model."

share|improve this answer

Test it both ways, especially if users are ever likely to encounter the situation where the DLL's have not been loaded up.

@Tristaan's answer goes into some good reasoning why you really ought to do it both ways.

That being said, in most performance or load tests we gradually ramp up the load because that's how it generally occurs in real life. So in most cases, the delays and issues with the initial loading of everything doesn't generally have a chance to affect the performance of a system in the real world as the load on the system increases.

About the only time a system will get suddenly slammed going instantly from nothing to full load is in a 'farm' situation which is running at high capacity and additional servers are spun up and brought online to help handle the load (something that can happen in a dynamically provisioned cloud environment). If that's something that could potentially happen in your case, then add that scenario to your tests as well.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.