There is no quantitative measurement for 'reasonable'. We will always disagree about what reasonable means. When using an application that is of no consequence to me, my tolerance level will be around 1s to 2s. However, my tolerance level will be much higher when I’m dealing with my bank (they have my money).
However, the measurement of response times, is of little consequence in performance testing. It’s is not a measurement indicative of how your production systems will behave. In my (not so humble) opinion, its just a distraction.
The ultimate goal of performance testing is to identify bottlenecks in applications before they manifest themselves in production. Its often cost prohibitive to replicate production systems running at internet scale in a performance environment. And if you spend all that money, you rarely get to reproduce the issues in production.
So lets focus on automated performance testing and what you should be measuring.
Your first objective is to ensure that all automated tests are running a sustainable load. This is actually easier said than done. I always try to keep the CPU running between 40% and 60%. You should now compare the throughput achieved and respective in one run with another. The actual results matter less. What we’re looking for is trends as we deploy new versions of the applications.
There are several “conditions” under which you could do performanc testing.
- Simulating Congested Networks. For internal applications, this is akin to using an application in another continent. For internet applications, this is aking to using DSL, or a mobile app in New York at lunch time.
- Simulating resource starvation. There are times in production when systems are running at their peak. It often means there is resource contention within the datacenter, network, database or applciation.
Simulating Congested Networks
When you have a user facing web application, one of the best things to do is to configure networking equipment to throttle bandwdith to that of congested networks, which is often around 700Kps to 1.5Mbps. Further configure the equipment to only allow 2 connections per IP address.
In this environment, the objective is to improve response times by looking at where applications spend their time. Poorly written AngularJS applications, for example, have the tendency to download over 1Mbps of assets and open more than 60 connections before it is ready for the user to use. This environment will exacerbate performance issues.
In order to measure whether the application performance has improved or deterioted, you’d run these tests using the same environment and compare the results between two runs. After a while you’ll discover a reasonable response time for the specific environment. Its response times has no relevance to the response times in other environments (e.g. production).
Simulating resource starvation
Another scenario is to remove the external limits, but to constrain the resources within the actual nodes. This implies reducing the memory, use slower disks and throttle network connectivity between web and database servers.
This often highlights issues that manifest themselves in production when the system is under load and resources are constrained.
Again, the response times is of less significance. What matters most is where the application is spending its time.
Conclusion
There is also a thought process that performance testing itself isn’t as useful as it use to be due to the scale at which applications run. One can only do a best effort with the expectation that you will most likely fail to prevent performance issues in production. Thus spend your precious time better understanding your production systems and identifying usage patterns that leads to performance bottlenecks.
The best way to measure performance from the users perspective in production, is by using something like Piwik, Google Analytics, Akamai RUM or Riverbed WebAnalyzer. If you use a CDN, but no analytics tool, then look at the origin response times.
