I am running single purpose performance tests on a daily basis. Currently, I run each test 3 times and average them out. But I get fluctuations from day to day which make it difficult to detect whether or not this is due to a code change.
What is a statistically appropriate way to measure and calculate performance?
The first thing you need to do is isolate whatever could be messing with your system. Make it as independent as possible. The fewer background processes, extra hardware and software, and even network traffic you can get, the better. Sometimes it's even a good idea to run these on a batch job that runs at night at a time when you know there's nothing else running.
The best statistical way that jumps in my mind is standard deviation. With a set of historical data, you should be able to acquire a mean and a stddev. If you get a result that is more than 3 or 6 stddevs from the mean (commonly known as 3-sigma or 6-sigma), you know you have something significantly different.
I'm sure you could simply pop the values in excel and get the stddev results.
In addition to what corsiKa said, you probably want to run a performance test more than 3 times. If you get one outlier because, say, a disk wasn't ready or some glitch occurred, you'll throw the average off significantly. For my team, when we want to test something like average time to operation completion, we'll run a hundred or so operations the way a user would and average that time. This is easiest if you have things automated, so you can schedule the tests to run for an hour or so at night and have results first thing in the morning.
