1

I'm load and stress testing a (AWS SQS) queue-based system. A single message injected into a queue kicks off the transaction's workflow. A message handler processes this message and then emits a message to another queue. Another handler handles that message and emits a message into the next queue. For the sake of simplicity, you can assume it's one continuous sequence of queues and message handlers. I monitor the transaction at the last queue and can record whether it completed successfully and on time.

I have a load testing system that increases the transaction-per-second (TPS) rate into the tested system until the number of failures increases beyond a certain level, at which point, I can claim I've found my max load.

Now for the stress testing part, I increase the TPS beyond the max load and see how the system reacts. Ideally, I expect the number of successful transactions to remain at the same max load with maybe a slight degradation, as more and more messages fail to complete on time. But what I'm seeing instead is a sudden drop-off in the TPS down to 0.

What seems to be happening is that the amount of time spent on all messages increases, instead of just the ones that arrive later. This delays all the transactions from completing on time.

In a synchronous system, it would probably happily serve all the traffic under its max load and simply start rejecting any beyond that. So while the number of failed transaction goes up, the number of successful transactions remain constant.

Am I stress testing my queue-based system correctly? If so, do you have any recommendations on how to make a stressed queue-based system perform more gracefully like one of a stressed synchronous system?

ps - I'm sorry I am unable to provide actual data and too much detail since this is a proprietary system. If there is something I'm missing, please ask and I'll provide it if I'm allowed.

Improve this question

1 Answer 1

Reset to default
1

Measure unprocessed transaction queue length

One approach would be to measure the queue length of unprocessed transaction and then learn where degradation and drop-offs occur.

Note that the unexpected drop offs may be a signal of the point where a normally functioning resource (memory, cpus, etc. becomes saturated and the queue starts to grow faster than be processed. Before this happens you might notice little to no slowness, but when it does the drop off can be severe.

Improve this answer
2
  • Yes I measure this to see which handler is not keeping up. and while this will allow me to rebalance resources so the workload is more evenly distributed and perhaps achieve a higher TPS, at some point the degradation becomes severe. my question is, is there a way to make the degradation less severe?
    – kane
    Jul 7, 2020 at 20:11
  • Add memory / cpu / re-write the code, parallelize. You can make the capacity higher, throughput greater and the memory usage more efficient. The actual 'cliffing' when it does occur may be hard to avoid because it is a severe change from 'always done instantly in how ever long it takes to do 1" to "done after all the other queued ones are done" - and the the queue in increasingly, the wait times will also be increasing. In other words job 1-5000 might take 1 seconds each but by 10,000 jobs could be waiting for 10 minutes due to a queue
    – Michael Durrant
    Jul 8, 2020 at 0:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.