I have a system with two logical threads, living on different machines and operating concurrently and asynchronously. First one receives data from an external system and puts them into DB. The latter receives action requests from the external system, and based on the data stored in the DB, it takes one decision or another, and then sends requests to third-party system.
The problem we have is because devs assumed that: data has been put in DB before the second thread receives request. In practice, it happened to me that there was no data in DB and the second thread took the wrong decision that impacts third party system severely.
I wonder how to test against such bugs in a systematic way?
The reason I have no found this defect before is that I took the same assumption the devs did: I waited for the data to be available in DB, before sending an action request. The way I found it was by (a) removing this wait condition from my automated test, plus (b) some coincidence: first thread was more overloaded than the latter and thus it delayed saving the data into DB.
My first ideas:
- Remove unreal wait conditions from tests
- Simulate such coincidences I described... But how?