Regression testing console apps

Question

I am wanting to set up some automated regression tests for some console apps. The apps are JSON transformers, so ideally what I'd like to do is pass the app some input files and compare the result to the spec output files, showing a diff and setting a failing exit code if they differ.

Does anyone have any suggestions of how I could do this? I've seen some good looking shell testing frameworks like BATS, but none have looked like they'd be easy to use with large output comparisons, ie, they would only tell you whether the output is a match, not providing any kind of detail let alone a diff. I'm not very familiar with GNU Diffutils at all, so perhaps I won't need much more than diff itself.

Answer 1

If all you want is pass-or-fail status on a series of scripts by comparing their output to a known-good output, you can easily setup GNU-Make to run the tests for you. Yes, to write a Makefile is to dabble in dark arts — but this sort of thing is exactly what make does and it will handle a ton of scenarios out of the box that most test frameworks would just be duplicating anyway.

Running tests

In your Makefile:

# Mark the test rule as not producing a matching file as output
.PHONY: test

# Run a series of tests
test:
    ./bin/program1 | diff - ./tests/expected_output1.json
    ./bin/program2 | diff - ./tests/expected_output2.json
    ./bin/program3 | diff - ./tests/expected_output3.json

Once the expected output is it place, make test should pass or fail depending on whether the programs are currently giving exactly the output they were expected too or not. (Note that if you use the .ONESHELL: option, also be sure to set the shell to fail on any failed exit code using .SHELLFLAGS = -e.)

Optional tests

If some of your tests are optional and you want to still run them but see the results of the failure without flunking the test run, you can use the - prefix. In the example above to mark program2 as allowed to fail, simply use:

test:
    ./bin/program1 | diff - ./tests/expected_output1.json
    -./bin/program2 | diff - ./tests/expected_output2.json
    ./bin/program3 | diff - ./tests/expected_output3.json

(Note this won't work with .ONESHELL: as in that configuration only the first line of the recipe can be prefixed, but you can add ||: to the end of the line to achieve the same thing with POSIX shell syntax.)

Optionally at run time you can choose to ignore errors and run all tests instead of aborting on the first failed one by using make -i test.

Generating expected test output

The above assumes you've generated the expected output by hand and saved it in the appropriate place. If the test output updates frequently you could automate that part too:

# Update test expected output
update_tests:
    ./bin/program1 > ./tests/expected_output1.json
    ./bin/program2 > ./tests/expected_output2.json
    ./bin/program3 > ./tests/expected_output3.json

Don't forget to list this target in the .PHONY: set. Once everything is setup, running make update_tests will regenerate the expected output, which you will probably want to commit to your repository. If the nature of your tests ever changes, run this target rule again and commit the changes to your repo.

DRY (Don't Repeat Yourself)

Better yet, if the tests are really as simple as the example above you could cheat and write both the tests and the expected output updater in one pass:

# Swap out the test action depending on a user overridable variable
UPDATE_EXPECTED := false
ifeq ($(UPDATE_EXPECTED),false)
    TEST_ACTION = | diff -
else
    TEST_ACTION = >
endif

test:
    ./bin/program1 $(TEST_ACTION) ./tests/expected_output1.json
    ./bin/program2 $(TEST_ACTION) ./tests/expected_output2.json
    ./bin/program2 $(TEST_ACTION) ./tests/expected_output3.json

Running make test will test as usual by substituting the action with "pipe to diff and compare with file". On the other hand when you want to update the test output you can run make UPDATE_EXPECTED=true test to make the action be a simple redirect that rewrites the output files instead.

Answer 2

One option would be to use your own test runner. I did this recently and now have:

do_test.sh
do_test_error_messages.sh

Include these files and then use like this, where "divisors", "amicables", etc. are the functions you want to test and then the last parameters are used by the function this way:

do_test setup $LINENO True 0
do_test divisors $LINENO failfast 1 1
do_test divisors $LINENO description="t2" 1,2 4
do_test divisors $LINENO 1 5
do_test divisors $LINENO 1,2,3 6
do_test divisors $LINENO 1,3 9
do_test divisors $LINENO 1 11
do_test divisors $LINENO 1,3,5 15
do_test divisors $LINENO 1,2,10,4,5 20
do_test sum_of_divisors $LINENO 1 1
do_test sum_of_divisors $LINENO 3 4
do_test sum_of_divisors $LINENO 6 6
do_test sum_of_divisors $LINENO 63 64
do_test amicable $LINENO 1 219
do_test amicable $LINENO 0 220
do_test amicable_list $LINENO 1,6,28 50
do_test amicable_list $LINENO 1,6,28,220 222
do_test amicable_list $LINENO 1,6,28,220,284,496 500
do_test amicable_list $LINENO 1,6,28,220,284,496 1000
do_test amicable_list $LINENO 1,6,28,220,284,496,1184,1210,2620,2924 3000
do_test sum_of_amicable_list $LINENO 35 50
do_test sum_of_amicable_list $LINENO 1035 500
do_test sum_of_amicable_list $LINENO 8973 4000

Full source code at:

https://github.com/durrantm/shell_test_runner