48

In my team, we use Selenium for automation of functional testing. We also do a lot of manual regression testing to make sure the appearance of the rendered web pages is ok. I would like to develop a framework, that would allow comparing screenshots of the "base" live web page, and the web page on the test server after applying the software update. I believe this will drastically reduce the amount of manual regression testing we need to do.

I would like the framework to be able to detect portions of web pages that were rendered differently from each other. An example of such functionality can be seen here. (first image in the article)

I have never worked with image processing and would need ideas on what algorithms are already publicly available that would help me with the task. From what I can see, the simple pixel-by-pixel comparison would not work well, because it does not account for the concept of elements.

Or am I approaching this problem from a completely wrong direction?

3
  • You can compare 2 images with md5sum command. it works fine for me. if your images are the same it will give you common result.
    – mortymacs
    Oct 22, 2014 at 6:06
  • Be careful going down this path, it's very messy and there are a lot of problems that need solving with image comparisons of pages. Different compression algorithms will have very subtle differences that to the human eye are non-existent, but to a machine that is able to look at each pixel, you will likely run into sporatic problems of images that look totally identical by practical standards coming up as not equal because there will exist single pixels that are only a few shades off from each other. To help solve this you should find a way to incorporate the idea of pixel tolerance
    – Julian
    Nov 9, 2015 at 20:33
  • How would you do this for all the different screen widths? Sounds really horrible to maintain. Feb 6, 2018 at 11:54

18 Answers 18

18

I'd take a look at Google QualityBots. It's generally used for comparing websites on multiple versions of Chrome, but looks very similar to what you are trying to do. I personally haven't had time to try it out myself mainly because of its use of EC2 machines. Other than that, it is open source.

3
  • 1
    This is terrific! Although I will not be using this system as it is designed, their approach to image comparison is exactly what I need. I'll be implementing it to work with our current Selenium based framework. Thanks!
    – osjak
    Jan 24, 2012 at 3:41
  • I'm pretty sure I would like to implement this in our selenium stack... +1
    – yoosiba
    Jan 24, 2012 at 16:22
  • @osjak - Did you get it implemented? I need to build something similar.
    – 9ikhan
    Apr 30, 2013 at 11:38
10

Imagemagick, a cross-platform imaging library and command line tool, has functions that can be used to compare images. A team I worked on circa 2012 used it with pretty good success to determine if two images differ. We had built a GUI with the same library that could pull the two images up side by side and highlight the differences for a human to decide whether the differences are significant.

1
  • Selenium has screenshot capabilities of the full webpage and individual elements. Imagemagick's language specific bindings can be used to do the screenshot comparison test. You can also use language specific image manipulation tools, but imagemagick is the faster and more feature reach than most image libraries.
    – SilentGuy
    Jul 3, 2019 at 20:56
3

I'm not sure if this would be what you're looking for, but, have you checked out sikuli? I've never used it myself, but, I've seen it recommended here a few times for the same type of tests.

7
  • I had the same thought. Here is a link to other SQA questions that mention sikuli: sqa.stackexchange.com/search?q=sikuli
    – user246
    Jan 23, 2012 at 14:03
  • sikuli is good to automate things based on images, but comparing images as osjak needs is not something you get out of the box. Look at blog.sikuli.org there you can see that image recognition in sikuli is used to find images on desktop, so you can use desktop areas as methods parameters. Calculating differences of images, marking differences is totally different topic. I would argue that sikulii is not much better to that task, than selenium (or basically language with which you use selenium).
    – yoosiba
    Jan 23, 2012 at 14:43
  • sikuli uses simple pixel-by-pixel comparison, from what I can see. It works for them, but I don't think it will help me. Thanks for the suggestion though.
    – osjak
    Jan 24, 2012 at 3:45
  • Well, unless you're comparing against exact match or some specific comparison algorithm or DOM structure, then Sikuli is useful for comparing images or any object/element on the screen/page represented by an image. The method to do this is to capture a reference element/object (say just a button) on the screen (crop it out from a screenshot) or if you need it relative to other objects, then capture a larger portion of screen or even the whole screen. Save the (cropped) capture as reference image. Then use Sikuli API in test to try and find if object exists anywhere on given screen.
    – David
    Oct 4, 2012 at 2:44
  • The Sikuli API has parameters for setting level of difference in finding images as well, though it might be less feature rich in how you specify differences compared to other tools or libraries.
    – David
    Oct 4, 2012 at 2:46
3

Here is a tool that will make visual comparison of different versions of your website dpxdt. Here is a video explaining how the tool works and how to use it: The Secret to Safe Continuous Deployment.

And finally, here is a guide how to implement it with Selenium.

3

Here's another one to throw into the pot, the BBC open sourced something that they use called Wraith, it's available on GitHub.

To steal directly from their readme:

What is it?

Wraith uses either PhantomJS or SlimerJS to create screen-shots of different environments and then creates a diff of the two images, the affected areas are highlighted in blue

Wraith Example

Requirements

You'll need either PhantomJS or SlimerJS, ImageMagick & Ruby 1.9.3 or greater. It's up to you to decide which browser engine you want to run it against.

It works on Windows, Linux and OSX.

2
  • It's probably worth noting that you will still need to manually check the diff to make sure it looks ok. You could trigger a failure based on the % difference but generally that is not very reliable.
    – Ardesco
    Jan 6, 2014 at 12:14
  • Similar project: github.com/Huddle/PhantomCSS
    – Brian Low
    Jun 20, 2014 at 4:25
2

If you are using python+selenium, there is a useful package, called needle:

It checks that CSS renders correctly by taking screenshots of portions of a website and comparing them against known good screenshots. It also provides tools for testing calculated CSS values and the position of HTML elements.

You can watch a short demo at Julien Phalip's talk from PyCon US 2014:

2

There should be opensource frameworks for image comparison out there. So you could use your existing Selenium stack to navigate through pages, then trigger given image processing framework.

I also worked with one guy, that did browser automation in AutoIt, and implemented in it image comparison. But later he switched to some external library for image comparison as it was fast.

For Selenium2 you could start with this link.

3
  • We're using fighting layout bugs. Quite tricky to set up - the out-of-the-box config is very sensitive in that it gives loads of false negatives. there are several parameters for reducing sensitivity though.
    – DuncN
    Jan 23, 2012 at 13:30
  • We also use a tool for the comparison between the 'base' image (oracle) & the newly rendered page. Our Programmers wrote it in-house based on some code they found on the web - I'll post a link when they get back to me.
    – DuncN
    Jan 23, 2012 at 13:33
  • This seems a bit too complicated for what I need. Thanks though!
    – osjak
    Jan 24, 2012 at 3:42
1

Just a quick question though - are you sure this is an avenue you want to go down? How many pages are you looking to check? Over a certain amount of pages, the time it takes to run the checks negates the value of the checks. This on top of tweaking the config to strip out false negatives. Add to this any dynamic data will throw out the comparison - one of our bugbears is googlemaps being slightly different each time the page is rendered

1

What you want to achieve is called automated visual regression / validation tests. Test automation tools such as Selenium or its wrappers (Protractor) only provide the automation part - and you are correct in wishing for visual validation of the page display. As you mentioned - you need the following elements:

  • Version control - to define a baseline that represents approved results and the sets of current results to compare against
  • Environment information - the ability to manage baselines per environment (OS, Web Browser type, viewport size - i.e. dimensions, device, etc.)
  • Integration with automation framework - essentially you want to keep your existing tests untouched - and only take a screenshot now and then
  • Image comparison - you want to avoid primitive pixel by pixel comparison as most of your tests will fail. You also want different comparison modes - i.e. - ignore changes that are undetectable to the human eye, or ignore changes that are proportionate to the page layout (picture a case when you change the language in the page - why should the test fail if everything behaved properly?)
  • Ability to define dynamic regions that should be ignored
  • support for iFrames
  • Last but not least: easy maintenance. If you made a legitimate change to the page - and all the tests failed (as they should) - you want to approve the change only once and have it automatically approved across all the steps. Otherwise you're facing the nightmare of going through all your tests and approving it.

As @user3210346 mentioned - pay a little money and use Applitools - as they are currently the only company that provides all of what I mentioned above.

I've had great experience with their product and they are very inclined to listen and add capabilities.

1

I agree with DuncN in many regards. Maybe a proof of concept would be helpful to try out before committing a great deal of time to it? There are paid services that have almost zero setup time and would tell you quickly whether the idea itself will be helpful like MogoTest, which I have used in the past.

It isn't free, but is fairly cheap and might be good to pull examples from for your own development should you choose to develop on your own.

1

Although it doesn't use screenshots, i.e. it is not pixel-based, you could use recheck-web (disclaimer: I work for retest, the company behind the project).

recheck-web is open source and based on recheck, which maps graphical user interfaces (and more, e.g., log files) to a special data model that enables you to detect visual regressions, but also non-visible changes such as CSS attributes or link targets. recheck-web, respectively, recheck essentially follow the principles of Golden Master Testing and Whitelist Testing.

It currently supports Selenium and offers a Java SDK, which looks as follows:

class MyRecheckWebTest {

    WebDriver driver;
    Recheck re;

    @Before
    void setUp() {
        driver = new ChromeDriver();
        // Use the default implementation.
        re = new RecheckImpl();
    }

    @Test
    void test() throws Exception {
        // Set the file name of the Golden Master.
        re.startTest( "my-golden-master-name" );
        // Do your Selenium stuff.
        driver.get( "https://my-url.com/" );
        // Single call instead of multiple assertions (doesn't fail on differences).
        re.check( driver, "my-test-step-name" );
        // Conclude the test case (fails on differences).
        re.capTest();
    }

    @After
    void tearDown() {
        driver.quit();
        // Produce the test report.
        re.cap();
    }

}

You can review test reports and maintain your Golden Masters either using recheck.cli (free) or review (paid).

0

With regards to Microsoft Expression, there is loads of software out there to help you do manual verification. Examples include:

https://browserlab.adobe.com/en-us/index.html
http://browsershots.org/

These are also great because they can be used for x-browser testing

0

There's also a simple old school method of comparing a given screenshot/image to a reference one in checking that they have the same MD5/SHA1 hash. That unfortunately likely may not work for screenshots of whole web pages. But it will at least work for comparing of individual images on a web page. Thought I'd point it out though.

0

I'd like to add PhantomFlow and PhantomCSS

Pros:

  • Good visualization report by D3 library. You can also generate the visualize user flow.
  • A action (click, type, select, etc.) can be added to the test by PhantomJS' javascript.
  • The screenshot quite stable from the PhantomJS headless browser.

Cons:

  • Only run for PhantomJS
  • Need to write javascript to make it run as your application user flow.
2
  • 1
    PhantomFlow and PhantomCSS won't work on pages with flex box elements. PhantomJS <v2 doesn't support them and both of these are built on them. Nov 21, 2015 at 1:54
  • PhantomCSS is no longer maintained: github.com/HuddleEng/….
    – beatngu13
    Mar 20, 2019 at 20:16
0

A new contender that works well is Ghostinspector There are free and paid versions.

You can set a tolerance and also target by css selectors, both great features.

enter image description here

1
  • How would you do this for all the different screen widths? Feb 6, 2018 at 11:54
0

I would not suggest using image comparison because exist much faster and easiest way to get precise results. Automation framework is a powerful tool for visual testing. It works using calculation, without CSS or image comparison.

Works super fast-here is on GitHub with examples of how to use it.

enter image description here

0

It sounds like what you are looking for is adding Visual Testing to your Functional Testing. Try out using Applitools. Here is a good article on Selenium Based testing using Applitools.

1
  • 3
    Broken link. It would be more valuable to summarize the article in your answer.
    – user246
    Dec 24, 2014 at 2:25
0

There is a project called "automated-screenshot-diff". You should give it a try.

It's a npm package used to compare images/screenshots. If you take a look at the examples folder you will see an example of how to take those screenshots dynamically with casperjs and then you can compare it using automated-screenshot-diff.

automated-screenshot-diff generates a perceptual diff from image A - image B and I think that is pretty much what the question is about.

Also... the project is able to generate a comparison report in JSON and HTML (to store as a Jenkins Artifact).

enter image description here

How it works?

Then automated-screenshot-diff will scan your screenshot's folder and calculate differences between your pre-production (stage) release and your production release. All generated image diffs will be in the same directory as your screenshots. If you don't know how to generate those screenshots, take a look at examples folder.

Just take 5 minutes to take a look at the project page.

2
  • Hi, Igor. Could you give some more information about how this project could help the poster and what it adds that hasn't been addressed in the other answers?
    – Kate Paulk
    Dec 16, 2013 at 12:33
  • @KatePaulk just edited my original answer to make it better. Jan 3, 2014 at 16:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.