What are the best approaches and what concerns are there about performance testing a web application that is hosted in the cloud?
Oh deja vu ...
I think that Larry Elison summed it up nicely when he stated that Cloud computing is just, servers and the internet. Video is here -> http://www.youtube.com/watch?v=UOEFXaWHppE
Performance testing "Cloud" applications should be tested as though you would test any existing web application. Specifically for performance you would want to testing a single instance and then how it scales out when you apply dynamic scaling to see if you get a linear performance improvement or not.
Apart from that the same performance testing tools and approaches and skills would apply, i.e. Loadrunner et.al.
Another challenge though would be that you won't have access to the performance logs, counters or configuration options that you would normally use when performance tesitng and tuning.
I think the key to your question is "web application". So we should not be directly concerned with which cloud you use or even latency.
Performance testing can mean different things to different people. I see it as:
Testing speed of pages
You need to know how the application performs in the view of users. Slow pages lower your revenue and customer satisfaction. To see the direct correlation, see these great performance testing statistics. Metrics like rendering time or response time are what your users care about. These measurements are what performance testing tools provide for you.
Secondly, and just as importantly, you should load test your entire site. Create realistic test scenarios/scripts that take actions on your site like actual users. Be especially sure to interact with your application in ways that stress your database because it is the most common bottleneck. Turn on the slow query log and review it to find any expensive joins or just bad SQL. Ramp up the load test volume from a small number to a large number. This allows you to see at what load the application starts to slow down or generate errors. Look for steep curves in response times that indicate "hitting a wall" for system performance. This can point you to a configuration limitation such as running out of database connections. Good cloud load testing tools include: LoadStorm, Browsermob, and Soasta.
Performance testing for web applications should focus on the user perspective. Be sensitive to errors and outliers of average response times.
Hope that helps. Lots of good resources on the same performance-testing.org site. Blogs, downloads, tool lists, articles, and industry news.
I have no idea about best approaches, but I can think of a few concerns to get discussion started:
- How much access do you have to the back end of the application? In theory where it's hosted shouldn't make any difference, but when you're dealing with cloud hosting you need to consider issues like your connection to the Internet, and the known capacity of the application's communication channels.
- What levels of access are you supporting with the app? Is it intended to be accessible via dialup? (A lot of apartments in the US have phone lines that might, maybe get 28k bandwidth on a good day). Does it need to be able to deliver streaming content or to cell phones?
- Do you need to consider throughput separately from response time? The kinds of things you'll want to look for and test are different if data needs to be pushed to a back end in real-time than if it's acceptable to maintain a queue that updates within an hour.
- This - again, in theory - shouldn't be a factor, but it still can bite you. Is there any possibility that heavy load on the app could take your cloud host down? If you're going to be testing against the hosted site, you probably want to take extra care to avoid effectively launching a denial of service attack against them :) (Don't laugh - the last time my team tried to load our web application to failure, we brought down the company's internal network, including the internal DNS and the web server... Effectively taking the whole company offline for a time. In our defense, none of us had much experience with load testing at the time, and we never did that again.)
There is a company called CloudHarmony.com that benchmarks and publishes all their information online for all the different cloud providers.
They basically run performance tests all the time, so it will save you the work of testing each provider.
Since 2009, we have performed 23,089,073 benchmarks in the cloud ~ cloudharmony.com/benchmarks
Your question is about testing a specific web app, in which case, the normal rules apply. The easiest, try out a tool like PageSpeed to give you feedback on the website.
If you are planning to test a cloud-based web application for its performance, some of the most important metrics are to work on storage, processing, bandwidth, and user accessing the system at any given point in time. Some of the other factors include elasticity of the cloud infrastructure for varying load variations, scalability, availability, reliability, and fault tolerance.
As long as it comes to testing techniques that can help you create a reliable web application include:
- Stress test
- Load test
- Browser testing
- Latency testing
- Targeted infrastructure test
- Failover test
- Capacity test
- Soak test
Besides, when it is about working on strategy, you can work on tactics that can help with cost savings, infrastructure access ease, reduction in cycle times, and choice of public, private, or hybrid cloud, among others. And if the application is hosted on some external cloud service provider, make sure you work on security, quality, reliability, latency, and bandwidth.