Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

The module I'm testing creates some graphical reports which include things like scatter plots and multipage line graphs. The bitmap will never be exactly the same between test runs, since some variables, like current timestamp labels, are drawn. Currently I'm testing this output by converting the PDF which is created into a bitmap, then doing a simple test of the percentages of red, green, blue, white, black, etc. pixels present in the entire bitmap. I'm mainly using primary colours in the graphics so it works reasonably well. I have an error tolerance (set to 0.01%) which allows minor changes to get through like the timestamp label.

This current solution is a bit fragile - e.g. when a valid change happens to the graphics, then the colour balance may change and so I'll need to update the percentages expected in the test. It feels like I'm not really testing the content of the bitmap, and rather I'm just testing that the test data generates the same bitmap every time -- i.e. its detecting a change rather than testing its doing the right thing.

I'm interested to hear if anyone thinks this is a good approach to testing graphics? Has anyone used any other approach to test graphics which are in the form of a bitmap and/or PDF. The app is a desktop application written in C# and WPF 4.0. I'm running tests using the built in unit test framework of Visual Studio 2010.

UPDATE: method of testing two images for similarity

I've found a better way of comparing two images is by comparing their color histograms via a chi squared test.

There's some more detail about the method here:

share|improve this question
Are you testing just the generation of the graphical report from a set of data, or are you also testing what happens before that, e.g. querying a database to extract the data that will be used to create the report? – user246 May 17 '12 at 14:26
@user246, yes I am testing what happens before the image generation in separate unit tests, e.g. reading and processing the data which is used to generate the graphics. So I'm viewing this particular test more like an integration test. – dodgy_coder May 18 '12 at 0:39
I am probably going to need to do something very similar to this as I need to test some heatmap images that one of our products is generating. I may try to go the route you mentioned using a chi squared test so I can compare images from different time periods and still know if they are similar enough that there are likely no bugs. – Sam Woods May 18 '12 at 23:47
up vote 5 down vote accepted

If the image is being generated by some process, you should be able to automate just about the entire process right up until generating the image with any data set you want.

Rather than testing every single data set being passed into whatever function generates the image, test the image generation the way you would approach testing almost any other unit: Test all of the boundaries and a range of valid and invalid inputs. These tests should be static and not have data that is changing all the time. If you are passing the exact same data into this function, the exact same image should be generated (except the timestamp which you mentioned).

It is true that you are testing it has changed instead of testing that it is doing the right thing. This isn't necessarily bad though, what you are doing is manual testing to verify it is valid and once you have valid results, recording that as your baseline. If the result changes down the road for the same input there are two possibilities - either the code was updated and it is a known change in which case you would need to update your baselines, or it is a bug in which case you file a bug and keep your existing baselines. I think there will always be that manual step of deciding whether the baseline should be updated or not.

share|improve this answer
Great points here, thanks. – dodgy_coder May 18 '12 at 0:42

For timestamps on the image, all other things being equal, you could just mask out that area to white, and then test as per a static image.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.