How to test the graphical contents of a bitmap or a PDF file?

Question

The module I'm testing creates some graphical reports which include things like scatter plots and multipage line graphs. The bitmap will never be exactly the same between test runs, since some variables, like current timestamp labels, are drawn. Currently I'm testing this output by converting the PDF which is created into a bitmap, then doing a simple test of the percentages of red, green, blue, white, black, etc. pixels present in the entire bitmap. I'm mainly using primary colours in the graphics so it works reasonably well. I have an error tolerance (set to 0.01%) which allows minor changes to get through like the timestamp label.

This current solution is a bit fragile - e.g. when a valid change happens to the graphics, then the colour balance may change and so I'll need to update the percentages expected in the test. It feels like I'm not really testing the content of the bitmap, and rather I'm just testing that the test data generates the same bitmap every time -- i.e. its detecting a change rather than testing its doing the right thing.

I'm interested to hear if anyone thinks this is a good approach to testing graphics? Has anyone used any other approach to test graphics which are in the form of a bitmap and/or PDF. The app is a desktop application written in C# and WPF 4.0. I'm running tests using the built in unit test framework of Visual Studio 2010.

UPDATE: method of testing two images for similarity

I've found a better way of comparing two images is by comparing their color histograms via a chi squared test.

There's some more detail about the method here:

• https://stackoverflow.com/questions/6499491/comparing-two-histograms
• https://stackoverflow.com/a/30606/507950
• http://www.itl.nist.gov/div898/handbook/eda/section3/eda35f.htm
• http://www2.lv.psu.edu/jxm57/irp/chisquar.html

Answer 1

If the image is being generated by some process, you should be able to automate just about the entire process right up until generating the image with any data set you want.

Rather than testing every single data set being passed into whatever function generates the image, test the image generation the way you would approach testing almost any other unit: Test all of the boundaries and a range of valid and invalid inputs. These tests should be static and not have data that is changing all the time. If you are passing the exact same data into this function, the exact same image should be generated (except the timestamp which you mentioned).

It is true that you are testing it has changed instead of testing that it is doing the right thing. This isn't necessarily bad though, what you are doing is manual testing to verify it is valid and once you have valid results, recording that as your baseline. If the result changes down the road for the same input there are two possibilities - either the code was updated and it is a known change in which case you would need to update your baselines, or it is a bug in which case you file a bug and keep your existing baselines. I think there will always be that manual step of deciding whether the baseline should be updated or not.

Answer 2

For timestamps on the image, all other things being equal, you could just mask out that area to white, and then test as per a static image.