I am trying to improve the tests for a bunch of renderers that generate 2D images, and I am looking for advice and ideas for testing that are both efficient and thorough but also branch friendly.
All renderers can be set up in a massive number of ways - there are at least 30 independent variables in the newest(simplest) renderers, and easily hundreds of independent variables in some of the more mature ones. All these are stored in one giant blob of a state class which is used by all renderers. (If you want any further detail that would help, just ask)
To date, tests have been written using the following pattern:
- Add a feature to the renderer
- Copy and paste a test in the relevant test file (copy and paste code contains setup, render, automated image comparison, dump on fail)
- Change the relevant variable and run
A human then checks the actual output is acceptable. If it is, the actual ouptut is copied to the expected output directory.
Problems at the moment:
- The expected output directory is stored in SVN in the trunk of our source (to allow easy branching). However this is bloating our repository and inflaming checkout times (1hour clean checkout) and disk usage (1gb and growing).
- Test code is too long and tests are unclear due to the copy and pasting
- Adding new tests that combines the test with all prior features is never done because it's unfeasible, so test coverage is poor
And here's my ideas so far:
- Compare image hashes instead of pixel by pixel image comparison, with some sort of 'AddImageToHashDatabase' program run by the human verifying test output.
- Then create (finite) lists of possible input for each variable and iterate over them all, testing every combination
And some other ideas from my team:
- Don't try to test everything, just write a set of tests that randomly test certain things and generally hit the 'known to be used' cases (keep doing the current thing)
- Move expected output to another repository
- Make each test test a combination of totally different things to reduce number of images needed
Any ideas for improving our testing, whether they're from prior experience or just ideas are greatly appreciated. Thanks!