10

For testing the visual appearance of a website the use of tools like selenium (functional surface test) is not sufficient because it doesn´t check the visual appearance and/or the CSS-Code directly. Different approaches are possible

  • Defining style guides
  • Syntax checking of CSS and HTML
  • DOM comparison
  • Screenshot comparison

How do you check the visual appearance of websites from a web-application?

I implemented a screenshot comparison tool in Java. Screenshots are generated by a framework based on selenium. I started with pixel by pixel comparison. Actually I split the screenshot into different parts (maybe 10x10 rectangles) and compare the histograms of corresponding parts. For the future I want to determine relevant parts (regions of interest) automatically. Maybe someone has ideas how to do that.

1

8 Answers 8

4

The most efficient tool for testing the appearance of a website is still the human eye and brain.

That said, if you want or need to automate, image comparison is probably the least effective method because even with fuzzy logic it can generate far too many false positives. All it takes is a change in hardware (a different monitor or video card - this will change such things as color saturation, image brightness and so forth, and that's before you consider changes to screen resolution), and a screenshot image comparison will fail.

If I had to do this, I'd look at first verifying the site by eye, then building file compare routines for the CSS and HTML on a per-page basis, using a saved baseline. This would avoid issues with different rendering across different browsers (best handled by eye for the initial verification) and cause a lot less work when changes are made (for each image, you'd need to define the regions of interest, how much variance you allowed, and where you allowed it. For a file comparison you really only need a way identify and strip things like timestamps which will always change - it's not affected by a change to the baseline file).

There are test tools that perform image comparisons: the ones I've tried haven't impressed me. They may meet your needs.

2
  • 2
    Maybe it´s reliable to test the appearance by human. But if you have to test more than 1.000 sites for regressions it is not feasible. I tested much tools like wraith etc. But the most use ImageMagick to compare images pixel by pixel and generate a diff-image. That doesn´t meet my needs too just as file compare. The comparison has to be done visual by image processing and without access the DOM.
    – MRae
    Jul 18, 2014 at 12:03
  • Is there a reason you need to do the comparison this way?
    – Kate Paulk
    Jul 21, 2014 at 11:17
1

Must say never had success with this using an automated approach. In the end always ended up using in cash terms more expert resource than cheaper eye ball resource.

The problem with the last project where I saw this tried was the environment and application changes just came too quickly to gain from unchanged repeated testing; despite the client thinking the system was not very volatile.

If I had many sites with lots of static display I might try a hybrid approach with high level automation to create a target issue list. But I wouldn’t set my sights very high.

1

As mentioned before, Applitools Eyes is a service which does exactly that.

No more pixel comparisons and histograms and thresholds. Applitools have proprietary algorithms which analyse the screen and determine whether a screenshot matches the baseline. And they do a very good job at it. If you really want, you can also set a "human" level for comparison, e.g., you can use "layout" instead of "strict" to make sure that the two screenshots have a similar structure but not necessarily have the same contents.

The very cool Robot framework mentioned in stephenb's answer is actually built over Applitools python SDK, and Applitools also have their own SDKs for most of the popular programming languages (Python, Ruby, Java, C#..).

You view your test results over the web, and Applitools also provide automatic maintenance of the test results which no other tools provides. So, for example, if you mark a region as a dynamic region, Applitools will automatically identify and mark that region as dynamic in other steps of the test, and even in other baselines (different form factors) for the same test.

0

We have started to trial this - http://code.google.com/p/qualitybots/ A project from the Google test team, it's quite a good blog to read.

Although at the moment, we heavily test using crossbrowsertesting.com and saucelabs. We split the testing of our websites on different browsers in to three different categories.

Functional - Ensuring key process and functions of the sites work. Automated.

Rendering - Visual Test using cross browser tools, although we will be using automation to take the screen shots, to speed up the process.

Usability - All manual I am afraid.

0

Visual regression testing is getting a momentum as far as I can see. I started to collect information and do some basic testing with some of them. Here are a couple of places/tools you could check:

2
  • I know Sikuli to automate some things at GUI via screenshots. But how I can use it for visual testing. Is there a way to fuzzy compare screenshots?
    – MRae
    Jul 22, 2014 at 8:38
  • when doing matching, similarity can be set-up: "The visual content of a region is evaluated by using methods like Region.find(), which looks for a given rectangular pixel pattern within the region. The matching content in the region has a similarity between 0 (not found) and 1 (found and it is per pixel exactly matches to the pattern). The find can be advised, to search with a minimum similarity, so that some minor variations in shape and color can be ignored. (sikulix-2014.readthedocs.org/en/latest/region.html#Region) Jul 22, 2014 at 8:42
0

Check out this tool https://github.com/NaviNet/Robot-AppEyes - a Robot Framework library that uses a cloud based provider to perform the 'human eyes' visual verification part.

The expected results / baselines are set using a Web GUI it even allows for verification of a region based on its element locator - this I found very cool!

Anyone using Selenium2Library can simply add the Robot library and call the verification during your existing tests.

0

We have recently launched http://backtrac.io project that also allows you to run set of screenshots and compare them. One of main features right now is ability to compare your production and staging environments. In this way you can see what is going to be changed very efficiently and in this way catch bugs early.

0

Automotion framework is a powerful tool for visual testing. It works using calculation, without css or image comparison. Works super fast https://github.com/ITArray/Automotion - here is on GitHub with examples of how to use it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.