Does it make sense to perform a "load test" on an app which is deployed to cloud with SLA= 99.9?

Question

Given that we have deployed an app in Azure which gives us 99.99% availability (on Virtual-Machines) based on its SLA (Service Level Agreement). To simplify the case, let's say that my app is available at least 99.9% of the time (about 9 hours in year) and if the down-time of service exceeds this value, then Azure will compensate the costs.

As you know, the goal of "load test" is to figure out when will the system break, however we know that this app will not break by the extrema load.

If it is the case, does it make sense to do a load test on such app with high availability ?

Answer 1

Yes. There is a difference between uptime and performance.

Your app might have a 100% uptime (thus Azure's SLA is covered), but your users might be dealing with a (terribly) slow app because of:

• the number of users
• a memory leak
• inefficient algorithms etc

Of course, performing load tests on a non-production environment comes with other considerations. For example, take into account the differences with the production machines (assigned memory etc) and how Azure handles scaling.

Answer 2

Yes, load testing and performance testing for a couple of reasons.
Being on a VM, there's no difference between it running there an on another physical machine. You can use too many resources (memory leaks are still an issue), or exhaust your ports.

Although the VM has a 99.99% uptime, do any of the applications in front of it have the same SLA? Is that SLA actually for the tier that you're on (there are a number of SLA's for different tiers? I've seen a number of cases where down time has not been covered by SLA, because the SLA of the actual component that failed was met, or because of a transient issue.

To test your auto-scaling rules. In my experience, auto-scaling doesn't always work, especially if you have any applications or services that are running that require an individual license key.

Answer 3

You would absolutely want to find bottlenecks in your application. For some basic examples: Java VM can be launched with heap size that is easily exceeded under load and the app crashes with OOM, Docker images can similarly be allocated insufficient memory; database access may be unoptimised and cause too many concurrent requests as to degrade performance or even cause data loss. Neither of this is solved by network throughput and compute resource scaling provided by the platform, and for the issues that can be solved by scaling you want to be at least somewhat sure that the scaling won't eat your budget too fast. Last but not the least, you also might want to check how fast does your system recover after an outage, meaning system as a whole, not the host VM recovery that is governed by SLA.

SLAs and scaling do help. You can definitely take it easier on some metrics when testing performance testing of cloud-hosted systems and concentrate on those that highlight bottlenecks. At least that was my takeaway working as QA on cloud-based projects.

Answer 4

Load testing your app will almost surely not cause an outage in Azure's Virtual Machine service. So you don't need to be concerned with their SLA.

If the load tests are being done in your production environment, I would be more concerned with any SLAs your end-users have on your product, as sufficient load could cause your app to have an outage.

Your app will eventually crumple under some form(s) of high load. It's more a question of what levels before that are you looking to gain information about (and why)