Testing of distributed and asynchronous processing

Question

We have camel based system domain-agnostic data channel framework which facilitate the raw data flow from an external data source , validate the raw data, enrich and standardize the data to ensure that data can be consumed by our system, determine the changes to the data and persist the data. All of these asynchronously and exposed as rest webservices. We need to develop a test framework for this system. Please suggest tools and design for this.

Answer 1

I have worked on systems which involved (Async Processing). The implementation of Asyc processing will be technically achieved through below examples

• Queue based processing (Service Broker / MQ Series)
• Custom Tracking tables (JMS etc..)

In all this cases, You can either validate by having

• Retry for fixed number of times with configurable timeouts to validate messages are processed. I have seen a similar case in file processing, file will be picked up but it will not be loaded completely (processing). In this scenario I wait and retry for a configurable number of times to validate data loaded
• Option#2 - If you have any tracking table you can run your validation once tracking table notifies messaged is processed
• Option#3 - Wait until all messages are cleared from the queues, directly query where the message queues where messages are stored intermittently to identify message status

Answer 2

If you are black box testing the system it would be very hard to directly test asynchronism. Even if you try some sort of white box testing, for example looking at queue sizes you'll have a difficult time testing since your response times will usually be too long- things will change until you understand that something has happened and react to it. Why don't you just test the system with varying loads ? light to heavy loads, varying ratios of sequential to random timing, varying data sizes and combinations of the above.