Our system takes an object from external system every time the object gets changed (change event), processes it and put it back processed into this external system. So, end-to-end testing is about making a change to an object on one end and watching the result on the other end. If there is a defect, it might be in our system or in other components with which we interact. Before the message arrives to our system, it goes through a number of queues, queue bridges, and a legacy system. After it is processed it is sent to a chain of Web services that perform additional processing.
Once I find a defect, I start isolating it. Gathering all information is hard because of three issues we have:
Distributed components, distributed logs. I record input test data, actual test output and then all intermediate queues messages, and Web service requests/responses. I take them from components' log files. Since messages are distributed between multiple components and machines, I work with an "airplane cockpit" of three LCD monitors: three SSH consoles open and tools for monitoring remote queues.
Parallel conversations. The whole our platform can process a number of messages concurrently, and end-to-end testing environment is shared between many testers so I must instantly monitor what’s going on with my message in the chain. I must not confuse my message with messages of other testers.
Complex interaction protocols. Interaction protocols between some components are more complex: they include multiple request/response cycles with conditional branches. I must put messages related to a single event in a sequence, to understand what happened after what.
So how do you handle such problems?
Share with me or, if you have patience to read more, comment on my solutions ideas.
Solution ideas
Getting messages in one place. All components (excluding legacy ones) use Apache log4j for logging messages. It is possible to configure log4j on all machines to sent log entries to a log viewer (e.g., Apache Chainsaw) on a tester's machine. The limitation is that this solution will not monitor messages that were put by external system on queues, but have not been read by our system yet.
Identifying messages for a single event. An event in our system is a single object change. Each event triggers a conversation in our system thus each conversation can be uniquely identified (e.g., by id of a changed object and the time of change). Including the conversation id in log entries makes filtering/grouping messages by conversation easier. This worked for my small academic project, but might be harder for large system we have here, correct?
Ordering messages in a sequence. JADE framework provides a sniffer that can "sniff" ongoing conversations and draw a sequence diagram (see image below). This worked for components that used the same framework to communicate each other, but our components are heterogeneous (queues, Web services) and use different transport protocols (e.g. HTTP). Is there a tool that can sniff and record such conversations?