2

The main goal is to be able to send some messages through a current live system and trace outputs all along the way, then replay those same messages in a newer updated version of the system and compare the outputs for regression testing.

I am currently struggling to figure out a good way to do this when the system interacts with one or more external systems such as web services, databases, FTP, etc ...

The main problem is that I might not always get the same results from the external systems even when using the same messages because the data in the database might have since changed, a file might have been moved or deleted from the FTP file location, or the web service spits out different results for each run regardless of the same message being passed in. In this scenario it's difficult to be able to do this test against the real web service, database, etc ...

Does anyone have any tricks or tools or anything that might allow me to do something like this?

  • send some messages through a current live system - You mean you want to test in PRODUCTION system? You surely have QA/integration testing system to run your tests? – Peter M. - stands for Monica Jul 7 '16 at 15:19
  • I'm referring to the current test environment that is running live. QA is constantly beating on the system and just slamming a new version in there is problematic if they want to test other parts of the system without the new changes affecting anything. To do this we typically setup a new server with the new version of the product installed. We run our tests on the new server and when everything is good we decommission the original server and make the new server the current live test system. We are trying to send some messages through the live test environment and compare outputs to new. – Brian T Hannan Jul 7 '16 at 15:25
1

We use that style of regression test where I work, too. It can be challenging to ensure that the system produces consistent results for the same input.

Ask whether you can run your tests at a lower level. It may be possible to record/playback inputs and outputs of specific subsystems, rather than using inputs and outputs of the system as a whole.

Consider whether some of the outputs can be consistent. If may not be possible to cause your test system to reproduce exactly what your production system did at a particular point in time, but there may be pieces of behavior that you can reproduce. For example, if the output consists of five fields, perhaps you can reproduce two of them. That may be better than nothing.

If the system supports multi-tenancy, use it to create a test environment on your live system. Consider an web application that supports multiple customers. Each user's data is in a separate silo so that the inputs for one user do not impact the outputs for a different user. You could create test user that always exists on both the live system and the QA system, and use that for your regression testing.

0

In this case what I've done is create a proxy server for these third party services so you can control the interaction.

Example:

An iOS application we have talks to an API that another team works on, which we don't control on our team, so we created a proxy server to imitate the responses from the API. We also made it so we can customize the response to get a specific reaction in the UI. This way we're not dependent on their service for testing since we're not testing their service but our iOS application.

It's a lot of work but worth it in these situations.

  • I believe you are referring to "mocking" which won't work here because I'm running the first set of messages through the live system which is unchanged and will be connecting to the actual web service(s). I grab the outputs from the live run and then I want to replay those same messages but this time instead of running it through the live system I will be running it through a test version the system. This test version ideally would connect to the actual web service(s) but there is no guarantee I will get the same data back in the response that I did when calling it from the live system. – Brian T Hannan Jul 7 '16 at 14:25
  • 2
    running the first set of messages through the live system which is unchanged and will be connecting to the actual web service - You mean you test in PRODUCTION system? You surely have QA/integration testing system, where YOUR system is live, but you can mock outside services? In QA, you can configure your system to either connect to vendor's PROD system (and have no control over return values) or to mock service, which you control. – Peter M. - stands for Monica Jul 7 '16 at 15:24
0

Not all tests have to be acceptance tests, some regressions are better tested with unit testing where you mock external systems

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.