Strategies for load testing in a Live or Dev Ops environment?

Question

I am working on a strategy/plan to load and performance test a web-based application hosted in Azure. The application integrates with a couple of other hosted apps that we do not host (i.e. do not have access to the database or source). A stand-alone test environment is difficult. So I am leaning towards doing load testing in our live environment.

What are good strategies for load testing in a live environment?

Answer 1

A few suggestions:

1. If you can mock or fake the responses of the integrated apps, you could avoid the need to test live. It does require more work to set this up, but you then have a non-production environment you can use and you will not find yourself inadvertently load testing the integrated apps.
2. If you must test in production, make sure you know the times when your application is getting the least use. This will give you the window in which you'll be least likely to disrupt customers.
3. Schedule with your providers and your partners. I can't stress this enough. Your load tests will generate a spike in traffic and could violate your hosting and/or partnership terms of service. I strongly suggest you discuss this with your lead/manager to make sure you have permission to create and perform load tests before you start.
4. Minimize side effects. By this I mean that you should not be creating real transactions if you can possibly avoid it - so if your app is a shopping app, you should be using a test sales channel and a test payment method to generate sales. If it's a booking system, make sure there's a test event to book so you don't sell out real events.
5. Start with a small proof-of-concept. Make sure you can take one virtual user through a set of requests and responses before you do anything else.
6. Have access to application logs. As well as the load test logs themselves, you'll want to be able to examine the application logs to find any issues that don't surface directly in the test. You may need to get someone else to pull them for you, but you will need to examine them. This includes the web server logs.
7. Where possible, test one thing only. If you need to load test your integrations to the third party apps, where possible do this in isolation via API. Similarly, test as much of your app as you can without touching the third party integrations. This makes it easier to track down bottlenecks in the systems.

There are other suggestions, but these would be my main focuses in your situation.

Answer 2

If you have Live environment only I'm afraid you're limited to testing in "dead" times like overnight or weekend. Also make sure your tests have proper setUp and tearDown logic in place so your test would leave the system in the original state (nothing added, nothing removed)

For DevOps (I understand it as running on machines with lower hardware specifications) running full-scale load test might not be the option, but there are few things you still can do:

• Perform integration testing under the load if there are any configuration issues you should be able to detect them
• Run Soak Tests to check application against obvious memory leaks
• External 3rd-party service calls can either be validated and assessed in terms of impact or mocked so your test would target only application parts that you developed and can control

More information: Performance Testing in a Scaled Down Environment. Part Two: 5 Things You Can Test

Answer 3

Load Testing in live.

Firstly come up with a plan, figure out how long you want to run for, what load, the models and what you want to measure overall.

Do you need let the businesses clients know as well as your dependencies? If you do, tell them advance of running your test, citing purpose, possible outcomes and outages.