Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

For testing the visual appearance of a website the use of tools like selenium (functional surface test) are not sufficient because it doesn´t check the visual appearance and/or the CSS-Code directly. Different approaches are possible

  • Defining styleguides
  • Syntax checking of CSS and HTML
  • DOM comparison
  • Screenshot comparison

How do you check the visual appearance of websites from a web-application?

I implemented a screenshot comparison tool in Java. Screenshots are generated by a framework based on selenium. I started with pixel by pixel comparison. Actually I split the screenshot in different parts (maybe 10x10 rectangles) and compare the histograms of correspondending parts. For the future I want to determine relevant parts (regions of interest) automatically. Maybe someone has ideas how to do that.

share|improve this question
    
Doesn't answer your question, but some interesting web comparison tools can be found in this question: sqa.stackexchange.com/questions/2459/… –  Niels van Reijmersdal Jul 18 at 15:45

7 Answers 7

The most efficient tool for testing the appearance of a website is still the human eye and brain.

That said, if you want or need to automate, image comparison is probably the least effective method because even with fuzzy logic it can generate far too many false positives. All it takes is a change in hardware (a different monitor or video card - this will change such things as color saturation, image brightness and so forth, and that's before you consider changes to screen resolution), and a screenshot image comparison will fail.

If I had to do this, I'd look at first verifying the site by eye, then building file compare routines for the CSS and HTML on a per-page basis, using a saved baseline. This would avoid issues with different rendering across different browsers (best handled by eye for the initial verification) and cause a lot less work when changes are made (for each image, you'd need to define the regions of interest, how much variance you allowed, and where you allowed it. For a file comparison you really only need a way identify and strip things like timestamps which will always change - it's not affected by a change to the baseline file).

There are test tools that perform image comparisons: the ones I've tried haven't impressed me. They may meet your needs.

share|improve this answer
    
Maybe it´s reliable to test the appearance by human. But if you have to test more than 1.000 sites for regressions it is not feasible. I tested much tools like wraith etc. But the most use ImageMagick to compare images pixel by pixel and generate a diff-image. That doesn´t meet my needs too just as file compare. The comparison has to be done visual by image processing and without access the DOM. –  MRae Jul 18 at 12:03
    
Is there a reason you need to do the comparison this way? –  Kate Paulk Jul 21 at 11:17

Must say never had success with this using an automated approach. In the end always ended up using in cash terms more expert resource than cheaper eye ball resource.

The problem with the last project where I saw this tried was the environment and application changes just came too quickly to gain from unchanged repeated testing; despite the client thinking the system was not very volatile.

If I had many sites with lots of static display I might try a hybrid approach with high level automation to create a target issue list. But I wouldn’t set my sights very high.

share|improve this answer

We have started to trial this - http://code.google.com/p/qualitybots/ A project from the Google test team, it's quite a good blog to read.

Although at the moment, we heavily test using crossbrowsertesting.com and saucelabs. We split the testing of our websites on different browsers in to three different categories.

Functional - Ensuring key process and functions of the sites work. Automated.

Rendering - Visual Test using cross browser tools, although we will be using automation to take the screen shots, to speed up the process.

Usability - All manual I am afraid.

share|improve this answer

Visual regression testing is getting a momentum as far as I can see. I started to collect information and do some basic testing with some of them. Here are a couple of places/tools you could check:

share|improve this answer
    
I know Sikuli to automate some things at GUI via screenshots. But how I can use it for visual testing. Is there a way to fuzzy compare screenshots? –  MRae Jul 22 at 8:38
    
when doing matching, similarity can be set-up: "The visual content of a region is evaluated by using methods like Region.find(), which looks for a given rectangular pixel pattern within the region. The matching content in the region has a similarity between 0 (not found) and 1 (found and it is per pixel exactly matches to the pattern). The find can be advised, to search with a minimum similarity, so that some minor variations in shape and color can be ignored. (sikulix-2014.readthedocs.org/en/latest/region.html#Region) –  Laurent Bristiel Jul 22 at 8:42

I must not agree with that: "The most efficient tool for testing the appearance of a website is still the human eye and brain." The most efficient way to do comparison of screenshots, whether it is of web pages, or 1000's of cups falling out of a plastic mold in an hour, on some factory, or anything else is by a software/machine. For industry purposes there are softwares which do screen shoots comparison for years (At least 15), and the good news is that there is a quite new startup called "applitools.com", which does exactly that on a very nice way. In short, you enter a line of code anywhere on your selenium code, telling it to take a screenshot. During the test it sends the screen shoot to their server, saves it, and by the end of the test, you go to your account, where you can see all the screen shoots of the test and approve them as passed or failed. Next time you will run the test, by the end of the test, you will know already if it had passed or not, and again you will get a link to where you can take a look at all these screenshots, old and new...approve them or not. You can adjust the tolerance of the comparison to a one pixel change, or make it less sensitive. In short, their service/app is the wet dream of any QA team testing UI or visible functional testing. Highly recommend!.

share|improve this answer

Check out this tool https://github.com/NaviNet/Robot-AppEyes - a Robot Framework library that uses a cloud based provider to perform the 'human eyes' visual verification part.

The expected results / baselines are set using a Web GUI it even allows for verification of a region based on its element locator - this I found very cool!

Anyone using Selenium2Library can simply add the Robot library and call the verification during your existing tests.

share|improve this answer

As mentioned before, Applitools Eyes is a service which does exactly that.

No more pixel comparisons and histograms and thresholds. Applitools have proprietary algorithms which analyse the screen and determine whether a screenshot matches the baseline. And they do a very good job at it. If you really want, you can also set a "human" level for comparison, e.g., you can use "layout" instead of "strict" to make sure that the two screenshots have a similar structure but not necessarily have the same contents.

The very cool Robot framework mentioned in stephenb's answer is actually built over Applitools python SDK, and Applitools also have their own SDKs for most of the popular programming languages (Python, Ruby, Java, C#..).

You view your test results over the web, and Applitools also provide automatic maintenance of the test results which no other tools provides. So, for example, if you mark a region as a dynamic region, Applitools will automatically identify and mark that region as dynamic in other steps of the test, and even in other baselines (different form factors) for the same test.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.