If all you want is pass-or-fail status on a series of scripts by comparing their output to a known-good output, you can easily setup GNU-Make to run the tests for you. In your Makefile: # Mark the test rule as not producing a matching file as output .PHONY: test # Run a series of tests tests test: ./bin/program1 | diff - ./tests/expected_output1.json ./bin/program2 | diff - ./tests/expected_output2.json Once the expected output is it place, `make test` should pass or fail depending on whether the programs are currently giving exactly the output they were expected too or not. This assumes you've generated the expected output by hand. If the test output updates frequently you could automate that too: # Update test expected output update_tests: ./bin/program1 > ./tests/expected_output1.json ./bin/program2 > ./tests/expected_output2.json Don't forget to add this rule to the `.PHONY:` set. Once everything is setup, running `make update_tests` will regenerate the expected output, which you will probably want to commit to your repository. If the nature of your tests ever changes, run that part again and commit it to the repo. Better yet, if the testes are really so simple you could cheat and write both the tests and the updater in one pass: # Swap out the test action depending on a user-overridable variable UPDATE_EXPECTED := false ifeq ($(UPDATE_EXPECTED),false) TEST_ACTION = | diff - else TEST_ACTION = > endif test: ./bin/program1 $(TEST_ACTION) ./tests/expected_output1.json ./bin/program2 $(TEST_ACTION) ./tests/expected_output2.json Running `make test` will test as usual by substituting the pipe to `diff`. On the other hand when you want to update the test output you can run `make UPDATE_EXPECTED=true test` to rewrite the output files instead.