Tell me more ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.
up vote 8 down vote favorite
4

In my team, we use Selenium for automation of functional testing. We also do a lot of manual regression testing to make sure the appearance of the rendered web pages is ok. I would like to develop a framework, that would allow to compare screenshots of the "base" live web page, and the web page on the test server after applying the software update. I believe this will drastically reduce the amount of manual regression testing we need to do.

I would like the framework to be able to detect portions of web pages that were rendered differently from each other. An example of such functionality can be seen here: http://expression.microsoft.com/en-us/dd565874 (first image in the article)

I have never worked with image processing, and would need ideas on what algorithms are already publicly available that would help me with the task. From what I can see, simple pixel-by-pixel comparison would not work well, because it does not account for the concept of elements.

Or am I approaching this problem from a completely wrong direction?

share|improve this question

8 Answers

up vote 5 down vote accepted

I'd take a look at Google Quality bots. It's generally used for compairing websites on mulitple versions of chrome, but looks be be very similar to what you are trying to do. I personally haven't had time to try it out myself mainly because of its use of EC2 machines. Other than that, it is open source.

Here is an article about it on Google's testing Blog

And the project page

share|improve this answer
1  
This is terrific! Although I will not be using this system as it is designed, their approach to image comparison is exactly what I need. I'll be implementing it to work with our current Selenium based framework. Thanks! – osjak Jan 24 '12 at 3:41
I'm pretty sure I would like to implement this in our selenium stack... +1 – yoosiba Jan 24 '12 at 16:22
@osjak - Did you get it implemented? I need to build something similar. – 9ikhan Apr 30 at 11:38
up vote 2 down vote

I'm not sure if this would be what you're looking for, but, have you checked out sikuli? I've never used it myself, but, I've seen it recommended here a few times for the same type of tests.

share|improve this answer
I had the same thought. Here is a link to other SQA questions that mention sikuli: sqa.stackexchange.com/search?q=sikuli – user246 Jan 23 '12 at 14:03
sikuli is good to automate things based on images, but comparing images as osjak needs is not something you get out of the box. Look at blog.sikuli.org there you can see that image recognition in sikuli is used to find images on desktop, so you can use desktop areas as methods parameters. Calculating differences of images, marking differences is totally different topic. I would argue that sikulii is not much better to that task, than selenium (or basically language with which you use selenium). – yoosiba Jan 23 '12 at 14:43
sikuli uses simple pixel-by-pixel comparison, from what I can see. It works for them, but I don't think it will help me. Thanks for the suggestion though. – osjak Jan 24 '12 at 3:45
Well, unless you're comparing against exact match or some specific comparison algorithm or DOM structure, then Sikuli is useful for comparing images or any object/element on the screen/page represented by an image. The method to do this is to capture a reference element/object (say just a button) on the screen (crop it out from a screenshot) or if you need it relative to other objects, then capture a larger portion of screen or even the whole screen. Save the (cropped) capture as reference image. Then use Sikuli API in test to try and find if object exists anywhere on given screen. – David Oct 4 '12 at 2:44
The Sikuli API has parameters for setting level of difference in finding images as well, though it might be less feature rich in how you specify differences compared to other tools or libraries. – David Oct 4 '12 at 2:46
show 2 more comments
up vote 2 down vote

Imagemagick, a cross-platform imaging library and command line tool, has functions that can be used to compare images. We're using it with pretty good success to determine if two images differ. We then have a GUI built with the same library that can pull the two images up side by side and highlight the differences for a human to decide whether the differences are significant.

share|improve this answer
up vote 1 down vote

There should be opensourced frameworks for image comparison out there. So you could use your existing Selenium stack to navigate through pages, than trigger given image processing framework.

I also worked with one guy, that did browser automation in AutoIt, and implemented in it image comparison. But later he switched to some external library for image comparison as it was fast.

For Selenium2 you could start with this one: http://code.google.com/p/fighting-layout-bugs/

share|improve this answer
We're using fighting layout bugs. Quite tricky to set up - the out-of-the-box config is very sensitive in that it gives loads of false negatives. there are several parameters for reducing sensitivity though. – DuncN Jan 23 '12 at 13:30
We also use a tool for the comparison between the 'base' image (oracle) & the newly rendered page. Our Programmers wrote it in-house based on some code they found on the web - I'll post a link when they get back to me. – DuncN Jan 23 '12 at 13:33
This seems a bit too complicated for what I need. Thanks though! – osjak Jan 24 '12 at 3:42
up vote 1 down vote

Just a quick question though - are you sure this is an avenue you want to go down? How many pages are you looking to check? Over a certain amount of pages, the time it takes to run the checks negates the value of the checks. This on top of tweaking the config to strip out false negatives. Add to this any dynamic data will throw out the comparison - one of our bugbears is googlemaps being slightly different each time the page is rendered

share|improve this answer
up vote 1 down vote

I agree with DuncN in many regards. Maybe a proof of concept would be helpful to try out before committing a great deal of time to it? There are paid services that have almost zero setup time and would tell you quickly whether the idea itself will be helpful like MogoTest, which I have used in the past.

It isn't free, but is fairly cheap and might be good to pull examples from for your own development should you choose to develop on your own.

http://mogotest.com/

share|improve this answer
up vote 0 down vote

With regards to Microsoft Expression, there is loads of software out there to help you do manual verification. Examples include:

https://browserlab.adobe.com/en-us/index.html
http://browsershots.org/

These are also great because they can be used for x-browser testing

share|improve this answer
up vote 0 down vote

There's also a simple old school method of comparing a given screenshot/image to a reference one in checking that they have the same MD5/SHA1 hash. That unfortunately likely may not work for screenshots of whole web pages. But it will at least work for comparing of individual images on a web page. Thought I'd point it out though.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.