Tell me more ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.
up vote 1 down vote favorite

Our company has started developing a new big web application. In a few months we testers will start integration testing, functional testing and regression testing both manual and automatic.
I am concerned about performance. As far as I know, no one will perform such testing. Actual wide-used web-apps have some bottlenecks, taking 30 to 120 seconds to load the heaviest pages.

When is the best time to start doing some performance testing, focusing just on the more important and/or more used pages?
My goal is to avoid letting the new web-app grow into a huge and slow resource monster that even timeouts automated testing, but I don't want to force any kind of premature optimization or putting pressure on the dev-team.

As a second question, how can I convince my boss that (I believe) a little performance testing is needed, since some actual projects are slow as hell? And, from a QA perspective, those load times are index of poor quality and poor service offered?

share|improve this question
Are you asking whether it makes sense to do performance testing at unit or integration level before measuring performance in end-to-end setup? – dzieciou Dec 17 '12 at 12:35
I'll have hands-on an end-to-end setup. My question is summarized in: better to start immediately (when the web app is "young" and can change very quickly) or wait when the app is more defined? – Alessandro Da Rugna Dec 27 '12 at 13:51
Customers didn't complain directly about performance, but some other problems they encountered and complained about are performance-related. – Alessandro Da Rugna Dec 27 '12 at 13:53

3 Answers

up vote 3 down vote accepted

The answer to this question is obvious: it depends. As an example, the customer complained on that the report generation pages takes a lot of time ~ 4 minutes to generate the report.

Our developers were not able to fix the performance without rewriting the legacy code from scratch. So they just added “ajax-like” loading indicator. Well… our customers were happy with this solution. And they become more happy when we added the “scheduled reports” feature that may run the reports in background.

The other pages has about 30 sec to 2 min load time, and it is still slow. But nobody cares, the customers are happy, the loading indicator fixed everything.

In other cases, like Amazon or Google the performance matters. Please take a look on this presentation from OreDev:

Web Performance (Honey Badger)

share|improve this answer
This is touching on "Percieved performance" which Ajax is helpful with. I still think there are important actual performance issues that should and could be found and resolved as well. – Sam Woods Dec 17 '12 at 17:49
Working on "perceived performance" is a nice idea in many situations, I didn't think about that! I'll propose this suggestion, thank you! – Alessandro Da Rugna Dec 27 '12 at 13:41
In the and I accepted your answer. Our customers complained about "performance" so now it gained attention. Thanks anyone for the great comments. – Alessandro Da Rugna Feb 13 at 10:59
up vote 2 down vote

The best time to start any testing is before any code is received.

Performance should be part of the Requirements and Architectural reviews/discussions that hopefully occur before coding begins. That's where you can have the most impact on performance, and can adequately plan for performance testing.

In your case, it seems pretty clear that none of the above has actually happened.

So in your case, the best time to start performance testing is probably "now".

share|improve this answer
Ok, I may prepare setup for testing now, implement some test cases. But I cannot test performance at component/integration level, because usually performance requirements are for end-to-end setup. Or are there projects that define this, e.g., response time for DAO component? – dzieciou Dec 17 '12 at 14:46
I agree with Joe. Even if you don't have clearly defined response times for individual components, stress testing them can definitely uncover unexpected resource consumption, flakiness and/or capacity issues. – Sam Woods Dec 17 '12 at 17:51
up vote 0 down vote

I believe I have some useful information about how/when to test performance, but first I wanted to touch on something that you said that stood out for me: "but I don't want to force any kind of premature optimization or putting pressure on the dev-team"

This indicates to me that the relationship you have with your dev team is perhaps not the most constructive. Your job as a QA engineer is to push for quality, but quality is the responsibility of everyone on your team. It shouldn't be a fight, the developers and project managers should be on board and behind anything that will increase quality. If that is not the case, perhaps there are bigger issues... I would absolutely push and put pressure on my team to perform performance testing. Developers, and the entire team should be proud of their work and how it performs is a huge part of any project. How can optimization be premature? Building and designing with performance (and security) in mind should absolutely be a priority and not left to clean up at the end (when it's already too late).

There are a few goals you may have from performance testing.

  1. Some SLA with your customers to perform within a certain time range and need to measure this.
  2. Predict how much hardware you need in your production environment under expected load. (Capacity Planning)
  3. Find and fix bottlenecks and bugs that manifest while under load.

1 and 2 can be useful to convince your team to do some performance testing, PM's will like 1, and PMs and Ops will like 2 and you and your dev team should like 3.

What I have found even through basic stress testing and monitoring of performance counters is that there is usually a lot of low hanging fruit, very simple fixes and tweaks that can make a huge difference in performance. People will find things like they are loading massive amounts of data into memory when they don't need to, or maybe there will be one specific query that is inefficient and slowing everything down, or some product or hardware settings are incorrect and causing problems, OR the hardware itself just isn't set up right - you've got great network throughput, plenty of CPU and only 2gb of memory, yet your app is memory intensive, or vice-verse.

share|improve this answer
1  
I disagree with the implication that optimization cannot be premature. It may never be too early to measure performance, but it is perfectly reasonable to decide that it is too early to use performance measurements to decide what to optimize. – user246 Dec 17 '12 at 18:21
1  
I may have stretched the term "optimize" a bit. The intention was to stress that thinking of performance from the very beginning is important, as Joe mentioned - part of design and planning. And again, performance testing can be done very early at a component level and still be effective, but you are right that actually making optimizations can wait, but be driven by the results of those tests. – Sam Woods Dec 17 '12 at 18:26

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.