Performance Testing a cloud application

Question

What are the best approaches and what concerns are there about performance testing a web application that is hosted in the cloud?

Answer 1

I have no idea about best approaches, but I can think of a few concerns to get discussion started:

• How much access do you have to the back end of the application? In theory where it's hosted shouldn't make any difference, but when you're dealing with cloud hosting you need to consider issues like your connection to the Internet, and the known capacity of the application's communication channels.
• What levels of access are you supporting with the app? Is it intended to be accessible via dialup? (A lot of apartments in the US have phone lines that might, maybe get 28k bandwidth on a good day). Does it need to be able to deliver streaming content or to cell phones?
• Do you need to consider throughput separately from response time? The kinds of things you'll want to look for and test are different if data needs to be pushed to a back end in real-time than if it's acceptable to maintain a queue that updates within an hour.
• This - again, in theory - shouldn't be a factor, but it still can bite you. Is there any possibility that heavy load on the app could take your cloud host down? If you're going to be testing against the hosted site, you probably want to take extra care to avoid effectively launching a denial of service attack against them :) (Don't laugh - the last time my team tried to load our web application to failure, we brought down the company's internal network, including the internal DNS and the web server... Effectively taking the whole company offline for a time. In our defense, none of us had much experience with load testing at the time, and we never did that again.)

Answer 2

Oh deja vu ...

I think that Larry Elison summed it up nicely when he stated that Cloud computing is just, servers and the internet. Video is here -> http://www.youtube.com/watch?v=UOEFXaWHppE

Performance testing "Cloud" applications should be tested as though you would test any existing web application. Specifically for performance you would want to testing a single instance and then how it scales out when you apply dynamic scaling to see if you get a linear performance improvement or not.

Apart from that the same performance testing tools and approaches and skills would apply, i.e. Loadrunner et.al.

Another challenge though would be that you won't have access to the performance logs, counters or configuration options that you would normally use when performance tesitng and tuning.

Answer 3

There is a company called CloudHarmony.com that benchmarks and publishes all their information online for all the different cloud providers.

They basically run performance tests all the time, so it will save you the work of testing each provider.

Since 2009, we have performed 23,089,073 benchmarks in the cloud ~ cloudharmony.com/benchmarks

Your question is about testing a specific web app, in which case, the normal rules apply. The easiest, try out a tool like PageSpeed to give you feedback on the website.

Answer 4

I think the key to your question is "web application". So we should not be directly concerned with which cloud you use or even latency.

Performance testing can mean different things to different people. I see it as:

1. Testing speed of pages

2. Load testing

You need to know how the application performs in the view of users. Slow pages lower your revenue and customer satisfaction. To see the direct correlation, see these great performance testing statistics. Metrics like rendering time or response time are what your users care about. These measurements are what performance testing tools provide for you.

I would recommend you analyze the performance of individual pages first. Use a tool like YSlow, webpagetest.org, Yottaa.com, or Load.in. These will show you the many aspects of speed for one page - connection time, when the page starts to render, when the document is fully loaded, and all of the ancillary files such as images, CSS, Javascript, and XML. Run performance tests on all of your important, high-traffic pages. Make improvements such as caching, compression, shrinking images, or using a CDN. Most of these tools will provide detailed suggestions for speeding up the page.

Secondly, and just as importantly, you should load test your entire site. Create realistic test scenarios/scripts that take actions on your site like actual users. Be especially sure to interact with your application in ways that stress your database because it is the most common bottleneck. Turn on the slow query log and review it to find any expensive joins or just bad SQL. Ramp up the load test volume from a small number to a large number. This allows you to see at what load the application starts to slow down or generate errors. Look for steep curves in response times that indicate "hitting a wall" for system performance. This can point you to a configuration limitation such as running out of database connections. Good cloud load testing tools include: LoadStorm, Browsermob, and Soasta.

Performance testing for web applications should focus on the user perspective. Be sensitive to errors and outliers of average response times.

Hope that helps. Lots of good resources on the same performance-testing.org site. Blogs, downloads, tool lists, articles, and industry news.

Answer 5

From my perspective, dynamic scaling has to be tested in advance to ensure that everything is good. The infrastructure's support cloud provider has to be used to get apple-to-apple comparison. If you test environment is not same as production, the integrity of data poses serious questions.