1

I am trying to write characterization tests, in order to quickly put a legacy system to a test harness.

I was not able to find many examples with production code. The examples I find are small programs.

An example by an author of Working Effectively with Legacy Code implements them as unit tests.

It's nice because it's human-readable and help you understand the code. However, I think it would work for small programs in the example, but the system I am trying to test is a pretty complex API client.

Other examples store the inputs and outputs in files and read them to compare the results with the 'snapshots'.

Some examples are VCR, Approval Tests, Golden Master Testing

I feel like this allows the test input to be generated, and suitable for testing a large set of inputs.

However, for some reason it feels more high-level than unit testing, and not very human-readable.

Are these tests supposed to be part of your unit test suite, or are they supposed to be complements to unit tests?

In other words, should I sample some characteristic test inputs and write characterization tests as unit tests, but using the actual output from the code to 'lock down' the existing behaviors, and trying to make it readable? Or should I treat characterization tests as a complement to unit tests? If so, what should I focus on in my unit tests?

3

Generally, characterisation tests are not an end state. They're a way to pin the current behaviour of untested legacy code, so that you can start making changes towards maintainability and testability with a reasonable level of confidence that the overall picture hasn't changed. They're not good tests, though, they don't:

  • tell you any of the behaviour is actually correct;

  • tell you where the problem is when a test starts failing (they're just detecting changes); or

  • tell you which parts of the current implementation are important.

So readability isn't particularly important in this case; once you have explored the behaviour more thoroughly, found the overlaps between the test cases (and which ones shouldn't have passed) and written higher quality tests that give you real confidence, you can get rid of most if not all of them.

The other use for characterisation tests is where the alternatives would be less readable, e.g. when you have code that outputs a report and you don't want to have to exhaustively check each part of it. Instead you generate the report, manually check it and then compare future reports to it to make sure nothing changes. Although this "snapshot"-style testing can be harmful at lower levels, enforcing "change detection" that actually prevents refactoring, at this level it's useful.

2
  • Thank you! I'm not sure if it's worthwhile to write snapshot tests for my team's software. Even though it has no unit tests, I am pretty familiar with the codebase. Also, I feel like writing snapshot tests for this system won't be trivial. It's an API client, and all the public functions just make calls for a third party API. Would you recommend writing snapshot tests first for any untested codebase, or does it sometimes make sense to skip snapshot testing and directly write unit tests?
    – user25409
    Sep 1 '20 at 19:09
  • @user25409 you often can't just jump straight to unit tests, because the code wasn't written in a way that enables unit tests and you can't change it with confidence in the absence of those higher level tests. Unit tests are in my view the least valuable to add because they're more coupled to implementation and will add friction to refactoring.
    – jonrsharpe
    Sep 1 '20 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.