I am running single purpose performance tests on a daily basis. Currently, I run each test 3 times and average them out. But I get fluctuations from day to day which make it difficult to detect whether or not this is due to a code change.

What is a statistically appropriate way to measure and calculate performance?

  • 1
    Maybe you should give more details about the program your testing. There are existing tools you may want to use.
    – Rivka
    May 7, 2011 at 21:40

2 Answers 2


The first thing you need to do is isolate whatever could be messing with your system. Make it as independent as possible. The fewer background processes, extra hardware and software, and even network traffic you can get, the better. Sometimes it's even a good idea to run these on a batch job that runs at night at a time when you know there's nothing else running.

The best statistical way that jumps in my mind is standard deviation. With a set of historical data, you should be able to acquire a mean and a stddev. If you get a result that is more than 3 or 6 stddevs from the mean (commonly known as 3-sigma or 6-sigma), you know you have something significantly different.

I'm sure you could simply pop the values in excel and get the stddev results.

  • 2
    What I'd like to add is: try to run each test much more times(for instance, 100-200) - more tests mean more measurement results which means less fluctuations in results.
    – alexb
    May 10, 2011 at 14:54
  • 1
    @alexb Certainly agree (as long you're trying to measure performance, not functionality.) In fact, running the tests many times can give you more than just a better indicator of what is average: it may yield slow-downs and memory leaks you wouldn't have otherwise found.
    – corsiKa
    May 10, 2011 at 15:31
  • Also consider extending test duration if possible, this will help average results and lower variance.
    – Rsf
    Dec 6, 2012 at 14:18

In addition to what corsiKa said, you probably want to run a performance test more than 3 times. If you get one outlier because, say, a disk wasn't ready or some glitch occurred, you'll throw the average off significantly. For my team, when we want to test something like average time to operation completion, we'll run a hundred or so operations the way a user would and average that time. This is easiest if you have things automated, so you can schedule the tests to run for an hour or so at night and have results first thing in the morning.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.