Compare screenshots of rendered web pages

Question

In my team, we use Selenium for automation of functional testing. We also do a lot of manual regression testing to make sure the appearance of the rendered web pages is ok. I would like to develop a framework, that would allow comparing screenshots of the "base" live web page, and the web page on the test server after applying the software update. I believe this will drastically reduce the amount of manual regression testing we need to do.

I would like the framework to be able to detect portions of web pages that were rendered differently from each other. An example of such functionality can be seen here. (first image in the article)

I have never worked with image processing and would need ideas on what algorithms are already publicly available that would help me with the task. From what I can see, the simple pixel-by-pixel comparison would not work well, because it does not account for the concept of elements.

Or am I approaching this problem from a completely wrong direction?

Answer 1

I'd take a look at Google QualityBots. It's generally used for comparing websites on multiple versions of Chrome, but looks very similar to what you are trying to do. I personally haven't had time to try it out myself mainly because of its use of EC2 machines. Other than that, it is open source.

• Here is an article about it on Google's testing Blog

• And the project page

Answer 2

Imagemagick, a cross-platform imaging library and command line tool, has functions that can be used to compare images. A team I worked on circa 2012 used it with pretty good success to determine if two images differ. We had built a GUI with the same library that could pull the two images up side by side and highlight the differences for a human to decide whether the differences are significant.

Answer 3

I'm not sure if this would be what you're looking for, but, have you checked out sikuli? I've never used it myself, but, I've seen it recommended here a few times for the same type of tests.

Answer 4

Here is a tool that will make visual comparison of different versions of your website dpxdt. Here is a video explaining how the tool works and how to use it: The Secret to Safe Continuous Deployment.

And finally, here is a guide how to implement it with Selenium.

Answer 5

Here's another one to throw into the pot, the BBC open sourced something that they use called Wraith, it's available on GitHub.

To steal directly from their readme:

What is it?

Wraith uses either PhantomJS or SlimerJS to create screen-shots of different environments and then creates a diff of the two images, the affected areas are highlighted in blue

Requirements

You'll need either PhantomJS or SlimerJS, ImageMagick & Ruby 1.9.3 or greater. It's up to you to decide which browser engine you want to run it against.

It works on Windows, Linux and OSX.

Answer 6

If you are using python+selenium, there is a useful package, called needle:

It checks that CSS renders correctly by taking screenshots of portions of a website and comparing them against known good screenshots. It also provides tools for testing calculated CSS values and the position of HTML elements.

You can watch a short demo at Julien Phalip's talk from PyCon US 2014:

• Advanced techniques for Web functional testing

Answer 7

There should be opensource frameworks for image comparison out there. So you could use your existing Selenium stack to navigate through pages, then trigger given image processing framework.

I also worked with one guy, that did browser automation in AutoIt, and implemented in it image comparison. But later he switched to some external library for image comparison as it was fast.

For Selenium2 you could start with this link.

Answer 8

Just a quick question though - are you sure this is an avenue you want to go down? How many pages are you looking to check? Over a certain amount of pages, the time it takes to run the checks negates the value of the checks. This on top of tweaking the config to strip out false negatives. Add to this any dynamic data will throw out the comparison - one of our bugbears is googlemaps being slightly different each time the page is rendered

Answer 9

What you want to achieve is called automated visual regression / validation tests. Test automation tools such as Selenium or its wrappers (Protractor) only provide the automation part - and you are correct in wishing for visual validation of the page display. As you mentioned - you need the following elements:

• Version control - to define a baseline that represents approved results and the sets of current results to compare against
• Environment information - the ability to manage baselines per environment (OS, Web Browser type, viewport size - i.e. dimensions, device, etc.)
• Integration with automation framework - essentially you want to keep your existing tests untouched - and only take a screenshot now and then
• Image comparison - you want to avoid primitive pixel by pixel comparison as most of your tests will fail. You also want different comparison modes - i.e. - ignore changes that are undetectable to the human eye, or ignore changes that are proportionate to the page layout (picture a case when you change the language in the page - why should the test fail if everything behaved properly?)
• Ability to define dynamic regions that should be ignored
• support for iFrames
• Last but not least: easy maintenance. If you made a legitimate change to the page - and all the tests failed (as they should) - you want to approve the change only once and have it automatically approved across all the steps. Otherwise you're facing the nightmare of going through all your tests and approving it.

As @user3210346 mentioned - pay a little money and use Applitools - as they are currently the only company that provides all of what I mentioned above.

I've had great experience with their product and they are very inclined to listen and add capabilities.

Answer 10

I agree with DuncN in many regards. Maybe a proof of concept would be helpful to try out before committing a great deal of time to it? There are paid services that have almost zero setup time and would tell you quickly whether the idea itself will be helpful like MogoTest, which I have used in the past.

It isn't free, but is fairly cheap and might be good to pull examples from for your own development should you choose to develop on your own.

Answer 11

Although it doesn't use screenshots, i.e. it is not pixel-based, you could use recheck-web (disclaimer: I work for retest, the company behind the project).

recheck-web is open source and based on recheck, which maps graphical user interfaces (and more, e.g., log files) to a special data model that enables you to detect visual regressions, but also non-visible changes such as CSS attributes or link targets. recheck-web, respectively, recheck essentially follow the principles of Golden Master Testing and Whitelist Testing.

It currently supports Selenium and offers a Java SDK, which looks as follows:

class MyRecheckWebTest {

    WebDriver driver;
    Recheck re;

    @Before
    void setUp() {
        driver = new ChromeDriver();
        // Use the default implementation.
        re = new RecheckImpl();
    }

    @Test
    void test() throws Exception {
        // Set the file name of the Golden Master.
        re.startTest( "my-golden-master-name" );
        // Do your Selenium stuff.
        driver.get( "https://my-url.com/" );
        // Single call instead of multiple assertions (doesn't fail on differences).
        re.check( driver, "my-test-step-name" );
        // Conclude the test case (fails on differences).
        re.capTest();
    }

    @After
    void tearDown() {
        driver.quit();
        // Produce the test report.
        re.cap();
    }

}

You can review test reports and maintain your Golden Masters either using recheck.cli (free) or review (paid).

Answer 12

With regards to Microsoft Expression, there is loads of software out there to help you do manual verification. Examples include:

https://browserlab.adobe.com/en-us/index.html
http://browsershots.org/

These are also great because they can be used for x-browser testing

Answer 13

There's also a simple old school method of comparing a given screenshot/image to a reference one in checking that they have the same MD5/SHA1 hash. That unfortunately likely may not work for screenshots of whole web pages. But it will at least work for comparing of individual images on a web page. Thought I'd point it out though.

Answer 14

I'd like to add PhantomFlow and PhantomCSS

Pros:

• Good visualization report by D3 library. You can also generate the visualize user flow.
• A action (click, type, select, etc.) can be added to the test by PhantomJS' javascript.
• The screenshot quite stable from the PhantomJS headless browser.

Cons:

• Only run for PhantomJS
• Need to write javascript to make it run as your application user flow.

Answer 15

A new contender that works well is Ghostinspector There are free and paid versions.

You can set a tolerance and also target by css selectors, both great features.

Answer 16

I would not suggest using image comparison because exist much faster and easiest way to get precise results. Automation framework is a powerful tool for visual testing. It works using calculation, without CSS or image comparison.

Works super fast-here is on GitHub with examples of how to use it.

Answer 17

There is a project called "automated-screenshot-diff". You should give it a try.

It's a npm package used to compare images/screenshots. If you take a look at the examples folder you will see an example of how to take those screenshots dynamically with casperjs and then you can compare it using automated-screenshot-diff.

automated-screenshot-diff generates a perceptual diff from image A - image B and I think that is pretty much what the question is about.

Also... the project is able to generate a comparison report in JSON and HTML (to store as a Jenkins Artifact).

How it works?

Then automated-screenshot-diff will scan your screenshot's folder and calculate differences between your pre-production (stage) release and your production release. All generated image diffs will be in the same directory as your screenshots. If you don't know how to generate those screenshots, take a look at examples folder.

Just take 5 minutes to take a look at the project page.

Answer 18

It sounds like what you are looking for is adding Visual Testing to your Functional Testing. Try out using Applitools. Here is a good article on Selenium Based testing using Applitools.