I'm load and stress testing a (AWS SQS) queue-based system. A single message injected into a queue kicks off the transaction's workflow. A message handler processes this message and then emits a message to another queue. Another handler handles that message and emits a message into the next queue. For the sake of simplicity, you can assume it's one continuous sequence of queues and message handlers. I monitor the transaction at the last queue and can record whether it completed successfully and on time.
I have a load testing system that increases the transaction-per-second (TPS) rate into the tested system until the number of failures increases beyond a certain level, at which point, I can claim I've found my max load.
Now for the stress testing part, I increase the TPS beyond the max load and see how the system reacts. Ideally, I expect the number of successful transactions to remain at the same max load with maybe a slight degradation, as more and more messages fail to complete on time. But what I'm seeing instead is a sudden drop-off in the TPS down to 0.
What seems to be happening is that the amount of time spent on all messages increases, instead of just the ones that arrive later. This delays all the transactions from completing on time.
In a synchronous system, it would probably happily serve all the traffic under its max load and simply start rejecting any beyond that. So while the number of failed transaction goes up, the number of successful transactions remain constant.
Am I stress testing my queue-based system correctly? If so, do you have any recommendations on how to make a stressed queue-based system perform more gracefully like one of a stressed synchronous system?
ps - I'm sorry I am unable to provide actual data and too much detail since this is a proprietary system. If there is something I'm missing, please ask and I'll provide it if I'm allowed.