The module I'm testing creates some graphical reports which include things like scatter plots and multipage line graphs. The bitmap will never be exactly the same between test runs, since some variables, like current timestamp labels, are drawn. Currently I'm testing this output by converting the PDF which is created into a bitmap, then doing a simple test of the percentages of red, green, blue, white, black, etc. pixels present in the entire bitmap. I'm mainly using primary colours in the graphics so it works reasonably well. I have an error tolerance (set to 0.01%) which allows minor changes to get through like the timestamp label.

This current solution is a bit fragile - e.g. when a valid change happens to the graphics, then the colour balance may change and so I'll need to update the percentages expected in the test. It feels like I'm not really testing the content of the bitmap, and rather I'm just testing that the test data generates the same bitmap every time -- i.e. its detecting a change rather than testing its doing the right thing.

I'm interested to hear if anyone thinks this is a good approach to testing graphics? Has anyone used any other approach to test graphics which are in the form of a bitmap and/or PDF. The app is a desktop application written in C# and WPF 4.0. I'm running tests using the built in unit test framework of Visual Studio 2010.

UPDATE: method of testing two images for similarity

I've found a better way of comparing two images is by comparing their color histograms via a chi squared test.

There's some more detail about the method here:

  • 2
    Are you testing just the generation of the graphical report from a set of data, or are you also testing what happens before that, e.g. querying a database to extract the data that will be used to create the report?
    – user246
    Commented May 17, 2012 at 14:26
  • @user246, yes I am testing what happens before the image generation in separate unit tests, e.g. reading and processing the data which is used to generate the graphics. So I'm viewing this particular test more like an integration test. Commented May 18, 2012 at 0:39
  • I am probably going to need to do something very similar to this as I need to test some heatmap images that one of our products is generating. I may try to go the route you mentioned using a chi squared test so I can compare images from different time periods and still know if they are similar enough that there are likely no bugs.
    – Sam Woods
    Commented May 18, 2012 at 23:47

2 Answers 2


If the image is being generated by some process, you should be able to automate just about the entire process right up until generating the image with any data set you want.

Rather than testing every single data set being passed into whatever function generates the image, test the image generation the way you would approach testing almost any other unit: Test all of the boundaries and a range of valid and invalid inputs. These tests should be static and not have data that is changing all the time. If you are passing the exact same data into this function, the exact same image should be generated (except the timestamp which you mentioned).

It is true that you are testing it has changed instead of testing that it is doing the right thing. This isn't necessarily bad though, what you are doing is manual testing to verify it is valid and once you have valid results, recording that as your baseline. If the result changes down the road for the same input there are two possibilities - either the code was updated and it is a known change in which case you would need to update your baselines, or it is a bug in which case you file a bug and keep your existing baselines. I think there will always be that manual step of deciding whether the baseline should be updated or not.


For timestamps on the image, all other things being equal, you could just mask out that area to white, and then test as per a static image.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.