# Is there a dancing pony on the screen? (Visual testing)

We currently have a rather rich subset of end-to-end tests for our AngularJS applications. And, we do get a lot of visual/layout/design issues on every single development iteration. Usually, we do not cover this kind of issues with end-to-end tests and testing the fixes manually. The reasoning was always - too hard, too fragile, too expensive to automate.

In some cases though, a style of an element or block on a page is really critical for an end-user and it would make sense to have an automated end-to-end test checking it. We try to check the colors, font styles and other CSS properties inside our Protractor tests, but it is still checking only these specific things and not an element or part of a page as a whole. Plus, the assertions themselves are not readable and easy to break, for example:

expect(somePage.someElement.getCssValue("background-color")).toEqual("rgba(255, 211, 205, 1)");


We are thinking of adding some sort of a visual component to our tests. Ideally, comparing screenshots of what we expect an element to look like and how it actually looks. But, there are a lot of open questions at this point:

• how reliable can it be - would it depend on the browser, resolution and other parameters of the test machine setup?
• is it possible to integrate with our existing Protractor test codebase or we need a separate specialized tool for that?
• test result reporting - how would we present the test results?

Have you done something similar and how did you address the above mentioned issues? Was it worth the cost of maintenance?

• Is there a reason why NOT to use one of visual testing tools you asked about (sqa.stackexchange.com/questions/29696/what-is-snapshot-testing) and try to develop one in-house instead? – Peter M. - stands for Monica Sep 21 '17 at 16:35
• @PeterMasiar yes, we are going to explore the tools suggested there (and the jest textual representations of components concept as well). Thanks! – alecxe Sep 21 '17 at 16:38
• Have you had a look at the Galen Framework? It follows a different but interesting approach. – beatngu13 Sep 21 '17 at 16:41
• @beatngu13 oh, wow, yeah, I like it, this may really work, will absolutely give it a try! Thanks again! – alecxe Sep 21 '17 at 16:43
• @PeterMasiar With Galen, you use a DSL to write a specification to test against, whereas golden master testing "derives" the expected results from the SUT. I think these are different testing methods, although you can use both for visual testing. Unfortunately, I haven't had the chance to work with Galen, but I'd love to hear how it performs (especially in terms of maintenance effort) compared with tools such as Depicted. So, alecxe, if you have a blog … ;-) – beatngu13 Sep 21 '17 at 17:45

how reliable can it be - would it depend on the browser, resolution and other parameters of the test machine setup?

I can answer this part. I am using Kantu for visual testing, and the tests do not depend on the resolution (the software and its image recognition algorithms account for that). So I can create a test on machine A and move it over to a very different machine B and it still works fine.

The problem with Kantu is that is works only inside Chrome (Chromium). So you can not use it for testing with Edge, Firefox etc.

For cross-browser visual testing we use Sikuli, which runs on the desktop. But while this works with all browsers, it is much more sensitive to resolution changes. So a test created on machine A does often not work on machine B.

• Oh yeah, I remember we had used sikuli extensively and it was a nightmare to support ever-changing UI, similarity coefficients, handling different systems and browsers. Not sure the cost of maintenance was worth it at the end. Thanks! – alecxe Sep 22 '17 at 14:22

You could use recheck-web for visual testing. It doesn't use screenshots (i.e. it is not pixel-based), but maps web GUIs to a special data model. This enables you to detect visual regressions, including non-visible changes such as CSS properties or hyperlink targets. It is built upon the principle of Golden Master Testing, so there is no need to write and maintain verbose specifications or complicated assertions.

how reliable can it be - would it depend on the browser, resolution and other parameters of the test machine setup?

For visual testing, you usually want your tests to catch differences among browsers, resolutions, etc. But recheck-web comes with a powerful ignore mechanism, so you can decide what changes you want to be notified about. For instance, you can ignore pixel differences up to a certain threshold or ignore font sizes while tracking font styles.

is it possible to integrate with our existing Protractor test codebase or we need a separate specialized tool for that?

Unfortunately, it currently only offers a Java SDK on top of Selenium and a Chrome extension. But it is open source, so you can easily get involved and contribute, e.g., suggestions for new SDKs, integrations, etc.

test result reporting - how would we present the test results?

Test results are persisted as .report files. You can view these (and maintain your Golden Masters) either using a CLI (free) or a GUI (paid). Here is an example:

(Disclaimer: I work for retest, the company behind the project.)