How do I automate regression testing against external web services, databases, etc ...?

Question

The main goal is to be able to send some messages through a current live system and trace outputs all along the way, then replay those same messages in a newer updated version of the system and compare the outputs for regression testing.

I am currently struggling to figure out a good way to do this when the system interacts with one or more external systems such as web services, databases, FTP, etc ...

The main problem is that I might not always get the same results from the external systems even when using the same messages because the data in the database might have since changed, a file might have been moved or deleted from the FTP file location, or the web service spits out different results for each run regardless of the same message being passed in. In this scenario it's difficult to be able to do this test against the real web service, database, etc ...

Does anyone have any tricks or tools or anything that might allow me to do something like this?

Answer 1

We use that style of regression test where I work, too. It can be challenging to ensure that the system produces consistent results for the same input.

Ask whether you can run your tests at a lower level. It may be possible to record/playback inputs and outputs of specific subsystems, rather than using inputs and outputs of the system as a whole.

Consider whether some of the outputs can be consistent. If may not be possible to cause your test system to reproduce exactly what your production system did at a particular point in time, but there may be pieces of behavior that you can reproduce. For example, if the output consists of five fields, perhaps you can reproduce two of them. That may be better than nothing.

If the system supports multi-tenancy, use it to create a test environment on your live system. Consider an web application that supports multiple customers. Each user's data is in a separate silo so that the inputs for one user do not impact the outputs for a different user. You could create test user that always exists on both the live system and the QA system, and use that for your regression testing.

Answer 2

In this case what I've done is create a proxy server for these third party services so you can control the interaction.

Example:

An iOS application we have talks to an API that another team works on, which we don't control on our team, so we created a proxy server to imitate the responses from the API. We also made it so we can customize the response to get a specific reaction in the UI. This way we're not dependent on their service for testing since we're not testing their service but our iOS application.

It's a lot of work but worth it in these situations.

Answer 3

Not all tests have to be acceptance tests, some regressions are better tested with unit testing where you mock external systems