Can existing tests and data effectively compare old and new systems?

Question

This idea was presented to me recently:

You have a complex legacy ETL system in Java in a TEST environment that no one understands completely. You write some tests that run it and compare the output with your expected outputs hardcoded into your tests.

But you also run your tests against a new rewrite of the system in a DEV environment and you compare the output of the legacy system in TEST with the output of the new system in DEV. If the rewrite was successful, the outputs should match.

So the question is, is this 2nd technique a good idea long-term? Short-term, it's convenient because you don't have to hardcode all your application's logic into your tests. You kindof skip understanding the business logic and you get a black-box comparison for free. But long-term, what's in DEV will be migrated to TEST and what's in TEST will be gone. New development will be in DEV. In the future running the suite will just compare this release's development with last release's development.

Someone proposed to me that this was still useful, but I'm not so sure. Something doesn't seem right to me. Some problems I see are:

1. You have to keep both TEST and DEV pointed at the same input data all the time.
2. You are losing the value of readable test specifications of behavior.
3. As soon as the legacy system is gone, you lose a lot of implicit behavior comparison from the old "source of truth" legacy system. You're now just comparing mostly against yourself.

Has anyone heard of examples of this or articles on it? I'd be interested if it has a name or hearing about the pros and cons. I couldn't find anything on Martin Fowler's site.

Answer 1

Yes this is a useful strategy in converting from one system to another as a one time event (it might span a couple of iterations but the idea is very short term).

You could effectively view this as BDD with 'half (the tests) already done'. You have the tests against data that works. You create a new back-end version and see if it does the same.

As I stated I consider this part of a conversion and it would not usually be practical to continue to run and maintain and compare both systems, post conversion.

In practice what you may often run into is that older test suites have grown over time, often through poor practices such as testing everything through the UI, not having good Unit and Integration tests, etc. Sometimes you may need to start with new tests that are more appropriate for the (probably) agile environment you are now in. Frequently, the real challenge here is explaining that to the business in terms they can relate to and ensuring you have executive buy-in. Otherwise you end up in a really bad place where you are saying 'we need much fewer ui tests' and the business looks at you and say 'no way, we care about the ones we have and won't drop any of them'. It's a long road to travel to show the power of well-written unit and integration tests to support 'less UI tests' but it is a road you must travel as real evidence trumps trying to change opinions.

Note on "As soon as the legacy system is gone, you lose a lot of implicit behavior comparison from the old "source of truth" legacy system" i think that means you need legacy systems tests for that currently implicit behavior in order to see if the new system meets those requirements.

At the end of the day I say:

Convert, Test, and go live with the new system. That just moves the challenge to having good tests and thats a good thing I think.

Answer 2

Comparing the old system to the new system given identical inputs and expecting identical outputs is a legitimate test. There are problems, however:

a) When the legacy system is retired, the Source of Truth is no longer available and comparisons are impossible. You'll have to be very comfortable that the systems work similarly enough to satisfy business requirements. One way to do this is to develop input and expected output data based on the legacy system and add them to the test suite of the new system.

b) The tests you develop and run against both systems are limited by your understanding of what is important in both systems. If a critical unwritten business rule is missed, there may be consequences.

One method to minimize the impact of a missed critical business rule is to run both systems in parallel with live data and log input and output. Users are brilliant at thinking of input combinations and use case scenarios I can't dream of. Let the legacy system run in the foreground, providing data to users and testers, and the new system in the background, providing data to testers only. Compare results. Tweak the new system until you have confidence the new system matches the old system.

This solution doesn't work very well if the data is confidential (ie. proprietary or credit card & banking data).

Answer 3

In my opinion, this approach can only work for very simple systems and for a very limited time.

Because, if the System B's output is exactly the same as System A's output it must mean the B is exactly the same as A and that is impossible for any non-trivial system.

So, as the system grows in complexity, the old tests become deprecated.

Of course, it is possible System B is a complete failure and it is underperforming, but I am assuming here that B > A (faster, more responsive, more features, more robust etc., or at least some of the above attributes).