Use of Standard Deviation and Percentiles in performance testing

Question

The following report is generated by Gatling (my favorite performance test tool). I am wondering how to make use of the standard deviation and percentile data that it produces.

I am planning to use the std. deviation to monitor if the times are too much spread across multiple releases. I am looking for how people use these statistics.

Answer 1

I use HP LoadRunner(fyi) so I have 90th percentile, but the question does not change.

I explain to managers consuming these reports:

The Std. Deviation number should be low. If your test is fairly consistent in load you want to see this number low because it means most response times are close in number.

The Mean is great but only accounts for what 50% of your users will experience. I tend to look at the 90th(95th) percentiles values, rather than mean, to ensure only those 10% wildcard calls are 'forgiven'.

I explain to technical people/developers consuming these reports:

The Std. Deviation number is too low/high. Lets use it to identify a potential bottleneck.

The 90th(95th) percentile is within/exceeds your SLA. Lets look at resource download times, contention points, or un-optimized stored procedures.

---

Hopefully this helps you. I try to keep it simple. As you very-well know, performance testing is an iterative process. If you find 1 issue, re-run and find the next! :D

Answer 2

From Microsoft's Key Mathematic Principles for Performance Testers. You likely do not have enough data for a statistically significant result.

1. You do not have 2 separate tests with 100 samples each.
2. If you believe that your measurements can bit modeled with a normal distribution pattern, your standard deviation should be less than half of your mean. Your mean and standard deviation are almost the same.