Is there a dancing pony on the screen? (Visual testing)

Question

Motivated by this talk about the accidental unexpected pony dancing on the screen.

We currently have a rather rich subset of end-to-end tests for our AngularJS applications. And, we do get a lot of visual/layout/design issues on every single development iteration. Usually, we do not cover this kind of issues with end-to-end tests and testing the fixes manually. The reasoning was always - too hard, too fragile, too expensive to automate.

In some cases though, a style of an element or block on a page is really critical for an end-user and it would make sense to have an automated end-to-end test checking it. We try to check the colors, font styles and other CSS properties inside our Protractor tests, but it is still checking only these specific things and not an element or part of a page as a whole. Plus, the assertions themselves are not readable and easy to break, for example:

expect(somePage.someElement.getCssValue("background-color")).toEqual("rgba(255, 211, 205, 1)");

---

We are thinking of adding some sort of a visual component to our tests. Ideally, comparing screenshots of what we expect an element to look like and how it actually looks. But, there are a lot of open questions at this point:

• how reliable can it be - would it depend on the browser, resolution and other parameters of the test machine setup?
• is it possible to integrate with our existing Protractor test codebase or we need a separate specialized tool for that?
• test result reporting - how would we present the test results?

Have you done something similar and how did you address the above mentioned issues? Was it worth the cost of maintenance?

Answer 1

how reliable can it be - would it depend on the browser, resolution and other parameters of the test machine setup?

I can answer this part. I am using Kantu for visual testing, and the tests do not depend on the resolution (the software and its image recognition algorithms account for that). So I can create a test on machine A and move it over to a very different machine B and it still works fine.

The problem with Kantu is that is works only inside Chrome (Chromium). So you can not use it for testing with Edge, Firefox etc.

For cross-browser visual testing we use Sikuli, which runs on the desktop. But while this works with all browsers, it is much more sensitive to resolution changes. So a test created on machine A does often not work on machine B.

Answer 2

You could use recheck-web for visual testing. It doesn't use screenshots (i.e. it is not pixel-based), but maps web GUIs to a special data model. This enables you to detect visual regressions, including non-visible changes such as CSS properties or hyperlink targets. It is built upon the principle of Golden Master Testing, so there is no need to write and maintain verbose specifications or complicated assertions.

how reliable can it be - would it depend on the browser, resolution and other parameters of the test machine setup?

For visual testing, you usually want your tests to catch differences among browsers, resolutions, etc. But recheck-web comes with a powerful ignore mechanism, so you can decide what changes you want to be notified about. For instance, you can ignore pixel differences up to a certain threshold or ignore font sizes while tracking font styles.

is it possible to integrate with our existing Protractor test codebase or we need a separate specialized tool for that?

Unfortunately, it currently only offers a Java SDK on top of Selenium and a Chrome extension. But it is open source, so you can easily get involved and contribute, e.g., suggestions for new SDKs, integrations, etc.

test result reporting - how would we present the test results?

Test results are persisted as .report files. You can view these (and maintain your Golden Masters) either using a CLI (free) or a GUI (paid). Here is an example:

(Disclaimer: I work for retest, the company behind the project.)