0

We're considering creating automated tests that confirm that specific items made it onto an SQS queue. Note that that sentence said "made" it - this would be a test that can be run on a live system. These tests would be run after changes are deployed to a given environment and the system is up and using those new bits.

From what I've found so far, it appears that AWS X-Ray may be able to look at recent items that made it onto the queue. Is X-Ray a sensical option? Do other, more-effective approaches exist?

Finally, is this a testing approach that tends to succeed in reality? Does strong guidance exist on how to test in a live system?

1

From your question I see that you want to make a smoke test on a production system. Testing on production could be very risky in some scenarios. So there are few questions that you should analyze:

  • Do you change the state of the system accessed only with specific user or they will be available to all prod users? If the data is available to prod users for me it sounds too risky.

  • How do you test the live system manually? Are you creating those items manually? If yes the how they are created, from some frontend publically available to all users or from some service to which the end users don't have direct access?

  • Think about some cleanup mechanism when you finish with the tests.

In my experience there were testers who wanted to change the global state of the production system for the purpose of the test an then revert this changes after the testing is done. The answer was totally NO. Now imagine that some real prod users take advantage of changed state and even worse => make nasty damage to the system or steal some private data.

My advice is to ask some tech lead/QA architect how something should be done properly. And as a general rule the lower environments like dev and stage should be more or less a mirror to the prod system. And if the tests pass on lower environments the chance to have a problem on prod is relatively low.

3
  • You describe nicely many of the things that I'm trying to answer as well. The more I think about it the more it feels like we need system designs that put testability as a first-class citizen. For example, in a production system, it would be of immense value if I could observe system history (even from moments ago) to confirm Givens, Whens, and resulting Thens. Jul 9 at 14:10
  • You can get the system history by adding loggers to your application, then you can search for a specific entry in the logs that satisfy your "Given" data. When and Then steps depends on the data from the logs, and you might be able to construct the logic for them. When you wrote history, I think about snapshot debugger in visual studio enterprise if the software was an azure application (but in your case is not).
    – chesthar
    Jul 10 at 6:52
  • Logging is a difficult option for test related needs, since we have multiple instances of processes on various machines. Getting the log data will require some tooling that gets results within a reasonable period of time. Jul 14 at 15:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.