Should I do performance testing of my web application having two instances behind a load balancer?

Question

My organization recently created a Performance testing environment. Our primary goal of performance testing is to catch performance degradation with future releases of our application. In Production, our web applications are deployed behind a load balancer, which is written and maintained by another team. To simulate Production environment, a decision was made to have two instances of each web application deployed in Performance environment, and execute performance tests against load balancer.

I feel that it is not an optimal decision, because we do not have control over load balancers, and my idea is to concentrate on testing performance of our application accessed directly via host name. Due to the fact that I lack experience with performance testing, I would like to understand arguments for and against these two different setups, and if I am correct, find necessary arguments to persuade management to change initial decision.

Answer 1

You need to decide on your goal:

• If you want to estimate your performance or capacity in production, you will want your test environment to mimic your production environment as much as possible.
• If your goal is to detect performance problems in your web application, you can probably get away with testing against a single instance. You won't necessarily be able to quantify how much performance degradation you will see in production, but that might be ok.

Answer 2

Performance testing environment need to mimic exact production steps. Please find my suggestions and questions

• Yes, You need to use loadbalancer (This is recommended). This would mimic exact production setup
• Do you have exact production hardware setup for performance testing ? Is it a scaled down version ? If it is a scaled down version then you need to understand How does the loadbalancer work, Is it IP based or http request based? If it is IP based then you need not worry much as there would be no routing involved (proxy handling), Load balancer also has logic to handle requests(Round robin, sticky sessions etc..)
• Alternatively if you can identify the amount of load, single instance can handle you can extrapolate how adding another instance would benefit
• What is the impact of load balancer on application logic, Is it just routing (dumb routing) ?

These are some questions to begin with. You can assess based on your application implementation and decide on next steps.

Answer 3

It depends on your application deployment scenarios. If it can be deployed as single instance you need to do performance checks of single-node deployment. If it can be deployed either as single instance or as 2x instance behind the load balancer - you need to check both. If your application isn't being the product itself and it's exposes some kind of service you need to do the performance testing under conditions as close to production as possible.

Summarizing all above: I would recommend to do performance testing of 2x instances and the load balancer in addition or instead of testing single node as for instance one of performance metrics is Latency which is "time until first response" and can be vital in case of complex deployments where network speed and routing hard or software configuration matter.

Answer 4

There are arguments for and against doing your testing against one or more than one server. Whether testing against one or more than one server however, I would still suggest including the load balancer in the tests. The load balancer configuration can have a major impact on performance and functionality of your site. It can, if configured incorrectly cause undesired behavior such as "session swapping" which is a good argument for executing tests against more than one server.

Executing your tests against a single server (but still behind a load balancer) can help you identify bottlenecks, and give you a good idea of how your application will perform. That is usually where I start. Sometimes it is also more practical, especially if that one server can handle a massive quantity of concurrent requests. It may be difficult to put together tests that will generate enough load to effectively test more than one server.

After executing those tests, you can then execute additional tests with more than 1 server behind a load balancer. In many circumstances doubling the servers will double your throughput, but depending on your infrastructure you may see diminishing returns by adding additional hardware. This is very important to understand if one of your goals for performance testing is capacity planning for future levels of user load. You may need to test with 2, or even more servers behind a load balancer to effectively plan server capacity. This exercise is more difficult if your hardware does not exactly match production hardware, but you can at least come up with some good educated guesses.

Answer 5

@oldbam: I have no idea about your loadbalancer configuration. Does it do persistence session (sticky loadbalancer) or just round-robin ?

Does your web application shares the user's sessions ? In high availability manner ?

All these questions will determine the type of performance test configuration:

Web application (high availability)

• this means the web server in each node will able to replicate the users' sessions among themselves.
• most likely the loadbalancer will be persistence to limit session swapping from one node to the other.
• if you are testing failover, then please use the loadbalancer.
• if you are testing the session replication between nodes, then direct all the traffics to one node and observe the session from the other.

Web application (non HA):

• loadbalancer might be round robin but most likely will be sticky as well to limit the invalid session cookies
• if you can doing capacity planning i.e., how much extra requests can be handled with one extra node and when the performance is degraded when there are too many nodes in the cluster, then use the loadbalancer.
• if you just doing simple stress test on the web application, a single node or multiple nodes via host file is alright. You don't need a loadbalancer to do that.