Visual Regression Testing

Question

For testing the visual appearance of a website the use of tools like selenium (functional surface test) is not sufficient because it doesn´t check the visual appearance and/or the CSS-Code directly. Different approaches are possible

• Defining style guides
• Syntax checking of CSS and HTML
• DOM comparison
• Screenshot comparison

How do you check the visual appearance of websites from a web-application?

I implemented a screenshot comparison tool in Java. Screenshots are generated by a framework based on selenium. I started with pixel by pixel comparison. Actually I split the screenshot into different parts (maybe 10x10 rectangles) and compare the histograms of corresponding parts. For the future I want to determine relevant parts (regions of interest) automatically. Maybe someone has ideas how to do that.

Answer 1

The most efficient tool for testing the appearance of a website is still the human eye and brain.

That said, if you want or need to automate, image comparison is probably the least effective method because even with fuzzy logic it can generate far too many false positives. All it takes is a change in hardware (a different monitor or video card - this will change such things as color saturation, image brightness and so forth, and that's before you consider changes to screen resolution), and a screenshot image comparison will fail.

If I had to do this, I'd look at first verifying the site by eye, then building file compare routines for the CSS and HTML on a per-page basis, using a saved baseline. This would avoid issues with different rendering across different browsers (best handled by eye for the initial verification) and cause a lot less work when changes are made (for each image, you'd need to define the regions of interest, how much variance you allowed, and where you allowed it. For a file comparison you really only need a way identify and strip things like timestamps which will always change - it's not affected by a change to the baseline file).

There are test tools that perform image comparisons: the ones I've tried haven't impressed me. They may meet your needs.

Answer 2

Must say never had success with this using an automated approach. In the end always ended up using in cash terms more expert resource than cheaper eye ball resource.

The problem with the last project where I saw this tried was the environment and application changes just came too quickly to gain from unchanged repeated testing; despite the client thinking the system was not very volatile.

If I had many sites with lots of static display I might try a hybrid approach with high level automation to create a target issue list. But I wouldn’t set my sights very high.

Answer 3

As mentioned before, Applitools Eyes is a service which does exactly that.

No more pixel comparisons and histograms and thresholds. Applitools have proprietary algorithms which analyse the screen and determine whether a screenshot matches the baseline. And they do a very good job at it. If you really want, you can also set a "human" level for comparison, e.g., you can use "layout" instead of "strict" to make sure that the two screenshots have a similar structure but not necessarily have the same contents.

The very cool Robot framework mentioned in stephenb's answer is actually built over Applitools python SDK, and Applitools also have their own SDKs for most of the popular programming languages (Python, Ruby, Java, C#..).

You view your test results over the web, and Applitools also provide automatic maintenance of the test results which no other tools provides. So, for example, if you mark a region as a dynamic region, Applitools will automatically identify and mark that region as dynamic in other steps of the test, and even in other baselines (different form factors) for the same test.

Answer 4

We have started to trial this - http://code.google.com/p/qualitybots/ A project from the Google test team, it's quite a good blog to read.

Although at the moment, we heavily test using crossbrowsertesting.com and saucelabs. We split the testing of our websites on different browsers in to three different categories.

Functional - Ensuring key process and functions of the sites work. Automated.

Rendering - Visual Test using cross browser tools, although we will be using automation to take the screen shots, to speed up the process.

Usability - All manual I am afraid.

Answer 5

Visual regression testing is getting a momentum as far as I can see. I started to collect information and do some basic testing with some of them. Here are a couple of places/tools you could check:

• Thoughtworks radar section on those tools: "This has given rise to a variety of visual regression testing tools, including CSS Critic, dpxdt, Huxley, PhantomCSS, and Wraith"
• and also Sikuli

Answer 6

Check out this tool https://github.com/NaviNet/Robot-AppEyes - a Robot Framework library that uses a cloud based provider to perform the 'human eyes' visual verification part.

The expected results / baselines are set using a Web GUI it even allows for verification of a region based on its element locator - this I found very cool!

Anyone using Selenium2Library can simply add the Robot library and call the verification during your existing tests.

Answer 7

We have recently launched http://backtrac.io project that also allows you to run set of screenshots and compare them. One of main features right now is ability to compare your production and staging environments. In this way you can see what is going to be changed very efficiently and in this way catch bugs early.

Answer 8

Automotion framework is a powerful tool for visual testing. It works using calculation, without css or image comparison. Works super fast https://github.com/ITArray/Automotion - here is on GitHub with examples of how to use it.