Testing approaches with temporal delays

Question

I am working with a system that has fairly complicated business logic. It has reasonably good unit, and integration tests.

We recently ran into a bug which we had missed. This bug only became apparent as certain business logic only applied after a number of days of a transaction being created.

We knew:

• Each component worked.
• User tests passed
• The system worked fine on day 0

But not:

• On day x after creation of items a rule would kick in that caused some items to fail.

We had specific tests for the rule, however we did not catch that it would apply to the items we were creating. These were a new type of item which the rule should not apply to.

So in tests everything was fine, and looked fine on production but after some time some users had problems.

This would seem a difficult thing to catch before hand. Now I have some tests in place that age the transaction and test that the logic is applied correctly we can stop regressions. But we endeavour to find bugs before production.

Are there any approaches that are used in such situations to try to catch things like this as part of a QA process?

Answer 1

This is the kind of potential problem that tends to be picked up by testers with a lot of in-depth and broad knowledge of the application when reviewing user stories/use cases/requirements - and it's very easy to miss.

Some of the methods I've used to try to catch this kind of problem include:

• Asking which rules are supposed to apply, and then asking for confirmation that other rules should not apply.
• A quick check of other related or similar types of items to work out what's different and what works the same way.
• Asking for explicit in scope/out of scope statements to use to guide your testing.
• Asking the team about any related functionality you know of and how the new feature interacts with it.

Honestly, this situation is one where tester serendipity comes into play: the methods I use are intended to try to trigger someone (including me) to think "Oh wait, I didn't think about X" and then investigate what should happen with X.

Since we can't always know in advance what everything should be, there are times when that's the best we can do.

Answer 2

Like Kate mentioned, these are easy to miss.

Many systems have tasks that kick off at certain times. Its good to ask for, or check, the cron tables to see those jobs. Then you can run a "mid-night" simulation test where you test those jobs.

Another example was a subscription billing system, where new tasks were fired at monthly intervals (charge the credit card, handle denial, suspend service). In that system, we ended up investing in a platform that advanced time on the server, and each test case had a time element in the verification.

Another example, we were testing a system that handed orders that expired. In that one, we were able to create back dated orders which should expire as soon as we entered them.

Also, for time/date based testing, make sure to cover leap days, and switching to daylight savings.

Answer 3

This looks like a problem with test data. It is often a difficult task to generate enough of test data in order to catch some defects which only appear together with heave workload, for example. Test data might have several parameters which could cause trouble:

• number of objects: e.g. having too many registered users may cause loss of older accounts
• unusual parameter(s) of a certain object: e.g. a 255-symbols long name or bank account with less than zero money
• age of an object: this is what caused your problem as far as I understand; etc...

Such occasions can be covered within the stress testing scope. You just have to define parameters which are likely to generate defects in case of reaching some borderline value(s), on beforehand.

Answer 4

From the perspective of an automated tester, these are the approaches I use:

1. Artificially age your test data to make the system think that records are older than they are

2. Create a test hook that triggers the time-based job on-demand

3. Ensure sure clocks/timers etc used by the application/s are injectable (maybe via a DI framework) or modifiable, so that time sensitive application behaviour can be faked - e.g. leap days, leap seconds midnight etc.

This sometimes requires modifying the product, or allowing tests to have direct access to backend persistence layers (which is often a bad thing for maintainability, depending on the stability of data schemas etc.) However, achieving reliable, independent, run-any-time tests is often a more important goal.

If you are dealing with complex system tests that want to check the interactions between all parts of the system, then this sounds like something that might best be handled by carefully monitored soak testing with a range of behaviours and timezones/states applied to different deployments.