13

We had a defect where in a situation the state of the application changed into something undesired. It has been fixed, we added a test:

  • Set up state
  • Action that triggered the wrong state change
  • Check state is still the same as setup
  • Wait some seconds
  • Check state is still the same and nothing changed

My question is: How long do we have to wait to make the test not generate false positives?

There are some situations where if the test environment restarts the services has a cold start of some seconds. The event-queue triggering the state change could be overloaded.

Theoretically the state could still change after a minute, due to the async nature of the services, but I don't want to add a test that always waits a minute or longer, hence my question. Are we aproaching this incorrectly or are there alternative options?

16

Great Question Niels.

Given that it will depend on the given situation I would consider approaching this as a data gathering exercise. I would investigate what is the normal distribution from many repeated runs. If 99% of the time it remains correct after 1 minute (i.e. after
1 minute doesn't then change again) and the business considers a 1% failure (and re-run) rate to be acceptable then I would use that. Same calculation for 10 second wait. Get the business to determine the right answer.

Usually I would hope to address this with a polling wait rather than a fixed wait, however in this case that doesn't help becuase of the potential for the answer to change from right to wrong after it has already been right initially.

One additional thought - this may belong to a class of issues that lend themselves to production monitoring rather than development testing. If this issue does happen in production, set up monitoring to capture the occurrence and frequency. With more data you may be able to formulate better plans ar approaches to mitigate the issue and/or change testing approaches to allow for it.

  • Thanks for the ideas. I think I will try to measure how long in a typical situation the transition could take. Take that as a baseline + some variance. Then do fail-fast polling for the negative situation, so that it doesnt want if the state is wrong. I agree that production monitoring could be a solution, I might want to add a A/B test monitoring, because in our situation this defect would store partial survey responses which researchers might not even notice. – Niels van Reijmersdal Oct 3 at 7:47
  • This answer expresses quite a lot of what I like about QA - most of the important, risky decisions are not on us! – Tim Oct 4 at 0:21
6

In this kind of situation on a development server I would be looking to get some information from the system itself to tell me when it has finished doing something.

I would expect the queuing system to be keeping records of what items have been on the queue, and what status they are now in.

The easiest way to do this is if you can find a key in the queuing system that allows you to link queue items in the database to the test accounts you're using. That way you could check whether any queue items for that account are not in a "DONE" state. If there are no items still pending, and the state in the app is as you were expecting, it's reasonably safe to assume that no new queue items will spawn on their own.

Without a key, you could check whether there are any queue items between two times that are not in "DONE". This method would be more prone to issues, so managing to identify a specific queue item as belonging to your tests is a better way of doing this.

If you wanted to be doubly certain, you could also create a system to check whether any additional queue items are created for the accounts that were used in the previous set of automated tests. You could then run this system some time after the main tests have finished, as a kind of paranoid "what if something is so wrong it creates another queue item after X minutes" kind of test. I'm not sure I would go that far though.

1

The event-queue triggering the state change could be overloaded.

I would expect an event-queue to be sequenced; that is, events pushed into the queue in order (1, 2, 3) are popped off the queue in the same order (1, 2, 3 and not 2, 1, 3).

Since the first event should not trigger a change, I would write two tests with the same canvas, each pushing a second event trigger a different change; the test would essentially go:

  1. Push event X (no change) and event W.
  2. Assert that nothing changes...
  3. Until change triggered by W occurs.

You may or may not want a slight pause between X and your Witness event.

The idea of using two tests with different effects is to ensure that you do not accidentally have overlap; that is, that X is not the event causing the effect expected of W.


Even better, though, would be an out-of-band signal.

This is something your development team should be able to provide. For example, you could have a Sequence event with an ID, where upon being processed the ID is reported out-of-bands -- not in-bands to avoid noise/perturbations.

Your tests can then become:

  1. Send Sequence(0).
  2. Wait until 0 is reported, signalling the system is started.
    • Or timeout, if the start-up takes too long.
  3. Send Command.
  4. Send Sequence(1).
  5. Wait until 1 is reported, signalling the command has been processed.
    • Or timeout, if waiting takes too long.
  6. Check that Command had the expected effect.

Such sequencing can really help test asynchronous systems, and especially reducing the flakiness of tests. It presumes that all asynchronous flows are synchronized, so the synchronization needs to circulate through all sub-systems before being reported.

Ideally, you would start the sequence at a random-number, and then increase it as you go.

If the application supports Heart Beats to monitor its responsiveness in production, then the Heart Beat should contain such a sequence number already to match query/response; ensuring it is well sequenced would then allow its usage for sequencing tests.

1

I would say that such cases are good candidates for white-box testing.

If developers are sure about what caused the issue to occur, the solution can be analyzed in terms of white-box techniques and tests can be designed to verify if supposed algorithm is indeed implemented in the way it was designed.

  • 1
    This would work if you control the whole software chain. In this case we get events from a third party. They changed their implementation which generated similar events for situations we did not want to process. – Niels van Reijmersdal Oct 3 at 7:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.