4

I have found performance targets (how long web pages take to respond to actions) difficult to introduce into organizations.

There seem to be are no clear definitions (in the industry) that are generally known to developers. It seems that it end up being a matter of preference.

This is complicated because we have little to no contact with out customers who visit out site from advertising and are then referred to other sites.

What are good ways to determine and use performance targets?

Ultimately I want benchmarks that I can use in automated performance tests. I will need to use average figures and also look at the distribution spread as historically one or two of a thousand queries takes abnormally long for a variety of reasons.

A frequent problem is that developers (naturally) want to test using the fastest setup and use high-speed wired connections that our users frequently don't have.

We currently use ruby on rails and capybara for ui testing but it does not seem to reflect real-world page load times.

See also Performance testing resources and Performance testing - a way in?

3
  • 1
    Defining acceptable response time have been part of usability studies and based on that they defined some targets: nngroup.com/articles/response-times-3-important-limits.
    – dzieciou
    Jan 3, 2016 at 9:08
  • Given that developers have no control over the latency in the line to the user, why would you want to introduce it into testing? Jan 4, 2016 at 7:45
  • So that when they make changes that affect performance it is noted before users experience it. They have control over efficient code, efficient database queries, efficient javascript, etc. Jan 4, 2016 at 21:31

4 Answers 4

4

Consider using A/B testing on your site to determine the impact of response time differences to your customers.

If done well, you will determine

  • If performance differences really matter
  • How much they matter
  • Thresholds after which your customers abandon your site
3
  • Does it mean some response time delay must be artificially introduced for instance to version B?
    – dzieciou
    Jan 3, 2016 at 13:59
  • Great idea. We do AB testing. I hadn't thought of it for this. It's usually content changes so it will take a little extra effort to use it for performance testing but it's a good idea. Jan 3, 2016 at 14:42
  • 1
    @dzieciou - Yes, that's one way. The other way is to "improve" A so that in the lab it's measurably faster than B. Then use A/B testing to find out if customers care, and if so, how much. I've done this in a server farm using smaller/slower servers compared to larger/faster servers with a load balancer in front. The web and application logs can help tell the results of the experiment, once analyzed. Jan 3, 2016 at 15:30
2

Measuring response times in realistic conditions

For most organizations, it is not worth the trouble to test your site against the variety of latencies and network speeds you see in production. This is particularly true if you are measuring not just the server response time but the total page render time. The latter depends on a lot of factors, including the speed of the client device and the browser type/version.

Instead, you can test with one (or a few) benchmark latency/speed combinations and then pay a third-party service like Keynote to measure response times in production.

Choosing a target response time

You care about response time because users like fast sites more than slow sites. If you want a specific target, you can use someone else's recommendation (like this) or you can base it on measurements of your own site using A/B testing.

2

Ideally acceptable response time ranges should be specified in SLA or NFR.

If they aren't - you need to make an assumption for how long end user will have to wait which may be minutes for i.e. in-house internal application or not more than few seconds for e-commerce websites where user will go away if the site operates slowly or down and will never come back.

In regards to developers testing on their own fast machines in intranet - load test needs to be executed in production environment (or staging environment which is exact replica of production). Any other test types don't make sense.

If you have already running application or service - you can choose "dead" time when real users load is minimal like 1 AM on Sunday.

See Load Testing Lessons to Learn from Black Friday 2014 article for few more real life examples.

2

There is no quantitative measurement for 'reasonable'. We will always disagree about what reasonable means. When using an application that is of no consequence to me, my tolerance level will be around 1s to 2s. However, my tolerance level will be much higher when I’m dealing with my bank (they have my money).

However, the measurement of response times, is of little consequence in performance testing. It’s is not a measurement indicative of how your production systems will behave. In my (not so humble) opinion, its just a distraction.

The ultimate goal of performance testing is to identify bottlenecks in applications before they manifest themselves in production. Its often cost prohibitive to replicate production systems running at internet scale in a performance environment. And if you spend all that money, you rarely get to reproduce the issues in production.

So lets focus on automated performance testing and what you should be measuring.

Your first objective is to ensure that all automated tests are running a sustainable load. This is actually easier said than done. I always try to keep the CPU running between 40% and 60%. You should now compare the throughput achieved and respective in one run with another. The actual results matter less. What we’re looking for is trends as we deploy new versions of the applications.

There are several “conditions” under which you could do performanc testing.

  1. Simulating Congested Networks. For internal applications, this is akin to using an application in another continent. For internet applications, this is aking to using DSL, or a mobile app in New York at lunch time.
  2. Simulating resource starvation. There are times in production when systems are running at their peak. It often means there is resource contention within the datacenter, network, database or applciation.

Simulating Congested Networks

When you have a user facing web application, one of the best things to do is to configure networking equipment to throttle bandwdith to that of congested networks, which is often around 700Kps to 1.5Mbps. Further configure the equipment to only allow 2 connections per IP address.

In this environment, the objective is to improve response times by looking at where applications spend their time. Poorly written AngularJS applications, for example, have the tendency to download over 1Mbps of assets and open more than 60 connections before it is ready for the user to use. This environment will exacerbate performance issues.

In order to measure whether the application performance has improved or deterioted, you’d run these tests using the same environment and compare the results between two runs. After a while you’ll discover a reasonable response time for the specific environment. Its response times has no relevance to the response times in other environments (e.g. production).

Simulating resource starvation

Another scenario is to remove the external limits, but to constrain the resources within the actual nodes. This implies reducing the memory, use slower disks and throttle network connectivity between web and database servers.

This often highlights issues that manifest themselves in production when the system is under load and resources are constrained.

Again, the response times is of less significance. What matters most is where the application is spending its time.

Conclusion

There is also a thought process that performance testing itself isn’t as useful as it use to be due to the scale at which applications run. One can only do a best effort with the expectation that you will most likely fail to prevent performance issues in production. Thus spend your precious time better understanding your production systems and identifying usage patterns that leads to performance bottlenecks.

The best way to measure performance from the users perspective in production, is by using something like Piwik, Google Analytics, Akamai RUM or Riverbed WebAnalyzer. If you use a CDN, but no analytics tool, then look at the origin response times.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.