2

Given the following example MongoDB document in a collection person, which goes through different processing steps:

{
  'status': 'Step 2 is finished'    
  'name': 'Some Name'
  'affiliation': 'Gang of Four'
}

Old software release: Documents were created in process step 1 without the field affiliation. The missing field was added later in process step 2.

After step 1:

{
  'status': 'Step 1 is finished'
  'name': 'Some Name'
}

After step 2:

{
  'status': 'Step 2 is finished'
  'name': 'Some Name'
  'affiliation': 'Gang of Four'
}

New software release: The entire document is created in step 1. Step 2 relies on the existence of the field affiliation and fails if it is not present.

The gap: A diff of the document schema between old and new release does not indicate a change, so no migration is built. When the new release is deployed, there are documents in the database which were produced by the old processing step 1, and are thus missing the affiliation field. Step 2 fails on these documents. The existing automated tests create test data dynamically though API calls, and so it is not possible to run them on 'old' data.

My ideas so far:

  • Generic tests, which search for documents in 'Step 1 finished' status and run the 'step 2 code' on them. Given a larger number of possible status and transitions, the number of necessary test scenarios would soon grow.
  • Base existing automated tests on static, raw database test data, which is versioned with the releases. This adds the burden of manual test data management which I would like to avoid.

What are good strategies to find these kind of gaps with automated tests, or to avoid them in advance?

1

What are good strategies to find these kind of gaps with automated tests ...

You could do a compare on the JSON string from one system to the other or do a SQL EXCEPT statement on the tables if you have access to those. This will tell you what is in one table but not the other.

... or to avoid them in advance?

Software versioning is used to indicate if you are looking at the "old" release or the "new" one. API versioning allows you to explicitly test the "old" one or the "new" one. You should always know which you are testing when you are testing it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.