When running automated performance tests, is it better to preload any DLL's which would slow the first run of a test? This would improve the consistency of the test results but would not reflect the worst case scenario.
How do you handle this issue in your performance tests?
I'm of two minds about this.
Essentially, two different kinds of testing require two different answers.
Test it both ways, especially if users are ever likely to encounter the situation where the DLL's have not been loaded up.
@Tristaan's answer goes into some good reasoning why you really ought to do it both ways.
That being said, in most performance or load tests we gradually ramp up the load because that's how it generally occurs in real life. So in most cases, the delays and issues with the initial loading of everything doesn't generally have a chance to affect the performance of a system in the real world as the load on the system increases.
About the only time a system will get suddenly slammed going instantly from nothing to full load is in a 'farm' situation which is running at high capacity and additional servers are spun up and brought online to help handle the load (something that can happen in a dynamically provisioned cloud environment). If that's something that could potentially happen in your case, then add that scenario to your tests as well.
If you look at any of the standard benchmarks available from TPC, SPEC, etc.. you will notice that all of them include a warm up period in the performance test, a time under full load, and then a ramp down interval. This ramp up period is designed to allow code which needed to be compiled upon first execution or libraries to be loaded to take place before the time under full load interval is entered into in order to measure optimal performance.
This ramp up/warm up period is established best practice in the commercial benchmark space, there are an awful lot of flat out poor/untrained performance testers in the market who simply go from 0 to 100% of load with few steps to allow for a warm up interval of any type. There is a market analogy for this called the "tea totaller." It goes something like this, "Take a Tea Totaller and a bottle of Vodka. Combine rapidly. You get total system shock. On the other hand, take a tea totaller, a bottle of Vodka and a thimble full of vodka at a time and you are likely to have a smoother ramp up to your test and you are also likely to find the exact spot at which your system melts down as load is increased....which is often at a point greater than the combine all at once model."
