4

Imagine a situation in performance regression testing where commit A is very deleterious to page performance, but does not trigger a failure - it's just under the level required to do that.

Commit B only degrades page performance a very small amount, but added on top of commit A, triggers a failure.

In this situation commit A is the one that requires attention, not commit B.

What are some techniques to deal with this issue?

(As an example - you are testing page load times and have a 3 second limit. Your baseline was a page load of 1 second. Commit A pushes it up to 2.9 seconds. Commit B pushes it up to 3.1 seconds)

Improve this question
5
  • what are you asking? It seems you know the code from A causes performance problems. Debug Code A. – squeemish Jul 8 '13 at 14:57
  • How does it need dealing with? Are you asking how to report this kind of problem to your development team? – Vince Bowdren Jul 8 '13 at 15:44
  • Have edited to make it a bit clearer what I'm describing. – dotcode Jul 8 '13 at 15:55
  • Is your question how to test for this possibility and how to identify where the problem is actually happened? And are you trying to automate this testing? – Daniel Jul 8 '13 at 17:11
  • Hi @Daniel - yes indeed, trying to automate this, and wondering how to identify where the problem happened. – dotcode Jul 8 '13 at 19:59
8

It's not exactly clear what question you are asking, but let me take a stab.

I would deal with it by creating a bug report. In it, I would mention what you are seeing in Commit A, and Commit B. I'd mention that the combination of the two pushes performance past the prescribed limits.

From a QA point of view, it's not important at all in which Commit the "blame" lies - only the fact that you now have a failure according to your current definitions. Let the developers debug both Commits and draw the proper conclusions.

Improve this answer
2

I assume your actual question is, "How do I narrow down which commits are responsible for a performance problem?" If you measure the performance after each commit, you can plot performance as a function of commits.

If performance is typically not a problem, you might choose to measure performance on a less frequent basis. When you observe a problem, you can use binary search (or something more sophisticated if you have the resources to run multiple test environments simultaneously) to determine where in the commit chronology the performance problem began and where it became a failure.

Improve this answer
2

I think the root problem here is the "automated way you detect perf degradations". A few ways to solve this:

  1. Have a human look at trend charts & spot "too large" perf degradation & file a bug. A human would have detected Commit A as the big culprit because doing from 1s to 2.9s is nearly 3x perf degradation. Yes, your company policy might be 3s max for the page, but a degradation of this size said something really destroyed previous awesome perf (relative to policy).
  2. If you desire to stick to automated "perf failure detection" then don't just have a hard limit of 3 seconds. Instead establish the 95th percentile page load time (for each page-type you perf test), and then on new perf tests if the (new) 95th percentile page load time is 30% higher than the benchmark then trigger a perf testcase failure.

You really want to use Percentile, not median or average & just peg it as some percentile like: 95th Percentile (or 99th, etc.). Percentile weeds out the outliers and gives you a fairly stable page load time run to run where as average & median can sometimes jump around a lot depending on how bad the outliers are.

Improve this answer
1

We're tackling a similar problem. Like Joe said, there's a certain point at which it's the developer's responsibility to troubleshoot it. That said, it's good to be able to test the concurrency performance of those sorts of things.

What we're trying is have tests that make the same call many times (perhaps tens or hundreds of times depending on the test case) and track page load times and point out major spikes or overall trends as failures as well as compare it to expected or acceptable times. I don't think any developer can accuse you of not doing enough testing there and it should be fairly easy to reuse that code for other commits you want to test.

Another tactic we're implementing in the same function is to vary to the content of the commits by type of data and size so we can also call out if variations in data show spikes or dangerous trends in load time.

Improve this answer
1

I think you are facing the problem like you made few changes in commit B and this makes an adverse effect on commit A. One of the tactic you can follow is performing regression testing with some kind of automated testing tools. Most of the tool do provide scheduled regression testing. Mind that regression testing is not at all re-testing.It continuously checks any adverse effect on the already working modules due to inclusion of new codes.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.