Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I am running single purpose performance tests on a daily basis. Currently, I run each test 3 times and average them out. But I get fluctuations from day to day which make it difficult to detect whether or not this is due to a code change.

What is a statistically appropriate way to measure and calculate performance?

share|improve this question
1  
Maybe you should give more details about the program your testing. There are existing tools you may want to use. –  Rivka May 7 '11 at 21:40

2 Answers 2

up vote 10 down vote accepted

The first thing you need to do is isolate whatever could be messing with your system. Make it as independent as possible. The fewer background processes, extra hardware and software, and even network traffic you can get, the better. Sometimes it's even a good idea to run these on a batch job that runs at night at a time when you know there's nothing else running.

The best statistical way that jumps in my mind is standard deviation. With a set of historical data, you should be able to acquire a mean and a stddev. If you get a result that is more than 3 or 6 stddevs from the mean (commonly known as 3-sigma or 6-sigma), you know you have something significantly different.

I'm sure you could simply pop the values in excel and get the stddev results.

share|improve this answer
2  
What I'd like to add is: try to run each test much more times(for instance, 100-200) - more tests mean more measurement results which means less fluctuations in results. –  alexb May 10 '11 at 14:54
1  
@alexb Certainly agree (as long you're trying to measure performance, not functionality.) In fact, running the tests many times can give you more than just a better indicator of what is average: it may yield slow-downs and memory leaks you wouldn't have otherwise found. –  corsiKa May 10 '11 at 15:31
    
Also consider extending test duration if possible, this will help average results and lower variance. –  Rsf Dec 6 '12 at 14:18

In addition to what corsiKa said, you probably want to run a performance test more than 3 times. If you get one outlier because, say, a disk wasn't ready or some glitch occurred, you'll throw the average off significantly. For my team, when we want to test something like average time to operation completion, we'll run a hundred or so operations the way a user would and average that time. This is easiest if you have things automated, so you can schedule the tests to run for an hour or so at night and have results first thing in the morning.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.