Testing asynchronous concurrent systems in a systematic way

Question

Problem

I have a system with two logical threads, living on different machines and operating concurrently and asynchronously. First one receives data from an external system and puts them into DB. The latter receives action requests from the external system, and based on the data stored in the DB, it takes one decision or another, and then sends requests to third-party system.

The problem we have is because devs assumed that: data has been put in DB before the second thread receives request. In practice, it happened to me that there was no data in DB and the second thread took the wrong decision that impacts third party system severely.

I wonder how to test against such bugs in a systematic way?

Proposed solution

The reason I have no found this defect before is that I took the same assumption the devs did: I waited for the data to be available in DB, before sending an action request. The way I found it was by (a) removing this wait condition from my automated test, plus (b) some coincidence: first thread was more overloaded than the latter and thus it delayed saving the data into DB.

My first ideas:

• Remove unreal wait conditions from tests
• Simulate such coincidences I described... But how?

Answer 1

When executing tests (either manually or automated), you must be careful about your assumptions.

Always explore if an application might be time/wait sensitive as you are conceiving and executing your tests.

• Always consider - what would a single user do (regarding wait period)?
• What would many users do?
• What happens if I go far faster than typical?
• What happens if I go far slower than typical?

Automation can help here, as long as you simulate closer to real-world conditions. With most tools, you can easily simulate many users arriving randomly, and waiting random periods of time.

Answer 2

Your ideas might be valid, but they use prior knowledge that might not be present for future bugs. Usually load and stress tests tend to catch similar issues, and so are long stability tests especially if they use somewhat above average load.