Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

In my team, we use Selenium for automation of functional testing. We also do a lot of manual regression testing to make sure the appearance of the rendered web pages is ok. I would like to develop a framework, that would allow to compare screenshots of the "base" live web page, and the web page on the test server after applying the software update. I believe this will drastically reduce the amount of manual regression testing we need to do.

I would like the framework to be able to detect portions of web pages that were rendered differently from each other. An example of such functionality can be seen here: http://expression.microsoft.com/en-us/dd565874 (first image in the article)

I have never worked with image processing, and would need ideas on what algorithms are already publicly available that would help me with the task. From what I can see, simple pixel-by-pixel comparison would not work well, because it does not account for the concept of elements.

Or am I approaching this problem from a completely wrong direction?

share|improve this question

13 Answers 13

up vote 8 down vote accepted

I'd take a look at Google QualityBots. It's generally used for comparing websites on multiple versions of Chrome, but looks very similar to what you are trying to do. I personally haven't had time to try it out myself mainly because of its use of EC2 machines. Other than that, it is open source.

share|improve this answer
1  
This is terrific! Although I will not be using this system as it is designed, their approach to image comparison is exactly what I need. I'll be implementing it to work with our current Selenium based framework. Thanks! –  osjak Jan 24 '12 at 3:41
    
I'm pretty sure I would like to implement this in our selenium stack... +1 –  yoosiba Jan 24 '12 at 16:22
    
@osjak - Did you get it implemented? I need to build something similar. –  9ikhan Apr 30 '13 at 11:38

Imagemagick, a cross-platform imaging library and command line tool, has functions that can be used to compare images. We're using it with pretty good success to determine if two images differ. We then have a GUI built with the same library that can pull the two images up side by side and highlight the differences for a human to decide whether the differences are significant.

share|improve this answer

I'm not sure if this would be what you're looking for, but, have you checked out sikuli? I've never used it myself, but, I've seen it recommended here a few times for the same type of tests.

share|improve this answer
    
I had the same thought. Here is a link to other SQA questions that mention sikuli: sqa.stackexchange.com/search?q=sikuli –  user246 Jan 23 '12 at 14:03
    
sikuli is good to automate things based on images, but comparing images as osjak needs is not something you get out of the box. Look at blog.sikuli.org there you can see that image recognition in sikuli is used to find images on desktop, so you can use desktop areas as methods parameters. Calculating differences of images, marking differences is totally different topic. I would argue that sikulii is not much better to that task, than selenium (or basically language with which you use selenium). –  yoosiba Jan 23 '12 at 14:43
    
sikuli uses simple pixel-by-pixel comparison, from what I can see. It works for them, but I don't think it will help me. Thanks for the suggestion though. –  osjak Jan 24 '12 at 3:45
    
Well, unless you're comparing against exact match or some specific comparison algorithm or DOM structure, then Sikuli is useful for comparing images or any object/element on the screen/page represented by an image. The method to do this is to capture a reference element/object (say just a button) on the screen (crop it out from a screenshot) or if you need it relative to other objects, then capture a larger portion of screen or even the whole screen. Save the (cropped) capture as reference image. Then use Sikuli API in test to try and find if object exists anywhere on given screen. –  David Oct 4 '12 at 2:44
    
The Sikuli API has parameters for setting level of difference in finding images as well, though it might be less feature rich in how you specify differences compared to other tools or libraries. –  David Oct 4 '12 at 2:46

There should be opensourced frameworks for image comparison out there. So you could use your existing Selenium stack to navigate through pages, than trigger given image processing framework.

I also worked with one guy, that did browser automation in AutoIt, and implemented in it image comparison. But later he switched to some external library for image comparison as it was fast.

For Selenium2 you could start with this one: http://code.google.com/p/fighting-layout-bugs/

share|improve this answer
    
We're using fighting layout bugs. Quite tricky to set up - the out-of-the-box config is very sensitive in that it gives loads of false negatives. there are several parameters for reducing sensitivity though. –  DuncN Jan 23 '12 at 13:30
    
We also use a tool for the comparison between the 'base' image (oracle) & the newly rendered page. Our Programmers wrote it in-house based on some code they found on the web - I'll post a link when they get back to me. –  DuncN Jan 23 '12 at 13:33
    
This seems a bit too complicated for what I need. Thanks though! –  osjak Jan 24 '12 at 3:42

Just a quick question though - are you sure this is an avenue you want to go down? How many pages are you looking to check? Over a certain amount of pages, the time it takes to run the checks negates the value of the checks. This on top of tweaking the config to strip out false negatives. Add to this any dynamic data will throw out the comparison - one of our bugbears is googlemaps being slightly different each time the page is rendered

share|improve this answer

I agree with DuncN in many regards. Maybe a proof of concept would be helpful to try out before committing a great deal of time to it? There are paid services that have almost zero setup time and would tell you quickly whether the idea itself will be helpful like MogoTest, which I have used in the past.

It isn't free, but is fairly cheap and might be good to pull examples from for your own development should you choose to develop on your own.

http://mogotest.com/

share|improve this answer

I'm the QA manager of Learni an education startup, and I'm a customer of applitools. Their brilliant app/sdk allows me to do exactly what you're describing, e.g adding one code of line, every time I want to take a screenshot, on the current selenium/appium code test you're already using, send it on the fly to their server, and when the test ends I go to their website where i see all the screenshot taken. Then I can approve what passed and what failed, so next time the same test runs, it already knows which test/ screen passes and which not. You may "tell" which parts of the screen to ignore (like dates which are changing), and it even knows to ignore this section on other screenshots (so you don't need to change it manually on 100's of screenshots). There are other many interesting features, but bottom line, there is no need to invent the wheel. Applitools almost literally, using state of the art technology, putting eyes in front of your tests screens, saving you the need to build such an app, and the need to watch it your self.
Good luck, thanks, Avi.

share|improve this answer

With regards to Microsoft Expression, there is loads of software out there to help you do manual verification. Examples include:

https://browserlab.adobe.com/en-us/index.html
http://browsershots.org/

These are also great because they can be used for x-browser testing

share|improve this answer

There's also a simple old school method of comparing a given screenshot/image to a reference one in checking that they have the same MD5/SHA1 hash. That unfortunately likely may not work for screenshots of whole web pages. But it will at least work for comparing of individual images on a web page. Thought I'd point it out though.

share|improve this answer

There is a project called "automated-screenshot-diff". You should give it a try: https://github.com/igorescobar/automated-screenshot-diff

It's a npm package used to compare images/screenshots. If you take a look into the examples folder you will see an examples of how to take those screenshots dynamicly with casperjs and then you can compare it using automated-screenshot-diff.

automated-screenshot-diff generates an perceptual diff from image A - image B and I think that is pretty much what the question is about.

Also... the project is able to generate a comparison report in JSON and HTML (to store as a Jenkins Artifact).

enter image description here

How it works?

Then automated-screenshot-diff will scan your screenshot's folder and calculate differences between your pre-production (stage) release and your production release. All generated image diffs will be in the same directory as your screenshots. If you don't known how to generate those screenshots, take a look at examples folder.

Just take 5 minutes to take a look into the project page.

share|improve this answer

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

    
Hi, Igor. Could you give some more information about how this project could help the poster and what it adds that hasn't been addressed in the other answers? –  Kate Paulk Dec 16 '13 at 12:33
    
@KatePaulk just edited my original answer to make it better. –  Igor Escobar Jan 3 at 16:18

Here's another one to throw into the pot, the BBC open sourced something that they use called Wraith, it's available on github:

https://github.com/BBC-News/wraith

To steal directly from their readme:

What is it?

Wraith uses either PhantomJS or SlimerJS to create screen-shots of different environments and then creates a diff of the two images, the affected areas are highlighted in blue

Wraith Example

Requirements

You'll need either PhantomJS or SlimerJS, ImageMagick & Ruby 1.9.3 or greater. It's up to you to decide which browser engine you want to run it against.

It works on Windows, Linux and OSX.

share|improve this answer
    
It's probably worth noting that you will still need to manually check the diff to make sure it looks ok. You could trigger a failure based on the % difference but generally that is not very reliable. –  Ardesco Jan 6 at 12:14
    
Similar project: github.com/Huddle/PhantomCSS –  Brian Low Jun 20 at 4:25

If you are using python+selenium, there is a useful package, called needle:

It checks that CSS renders correctly by taking screenshots of portions of a website and comparing them against known good screenshots. It also provides tools for testing calculated CSS values and the position of HTML elements.

You can watch a short demo at Julien Phalip's talk from PyCon US 2014:

share|improve this answer

Here is a tool that will make visual comparison of different versions of your website dpxdt. Here is a video explaining how the tool works and how to use it: The Secret to Safe Continuous Deployment.

And finally, here is a guide how to implement it with Selenium.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.