Motivated by this talk about the accidental unexpected pony dancing on the screen.

We currently have a rather rich subset of end-to-end tests for our AngularJS applications. And, we do get a lot of visual/layout/design issues on every single development iteration. Usually, we do not cover this kind of issues with end-to-end tests and testing the fixes manually. The reasoning was always - too hard, too fragile, too expensive to automate.

In some cases though, a style of an element or block on a page is really critical for an end-user and it would make sense to have an automated end-to-end test checking it. We try to check the colors, font styles and other CSS properties inside our Protractor tests, but it is still checking only these specific things and not an element or part of a page as a whole. Plus, the assertions themselves are not readable and easy to break, for example:

expect(somePage.someElement.getCssValue("background-color")).toEqual("rgba(255, 211, 205, 1)");

We are thinking of adding some sort of a visual component to our tests. Ideally, comparing screenshots of what we expect an element to look like and how it actually looks. But, there are a lot of open questions at this point:

  • how reliable can it be - would it depend on the browser, resolution and other parameters of the test machine setup?
  • is it possible to integrate with our existing Protractor test codebase or we need a separate specialized tool for that?
  • test result reporting - how would we present the test results?

Have you done something similar and how did you address the above mentioned issues? Was it worth the cost of maintenance?

  • Is there a reason why NOT to use one of visual testing tools you asked about (sqa.stackexchange.com/questions/29696/what-is-snapshot-testing) and try to develop one in-house instead? – Peter M. Sep 21 '17 at 16:35
  • @PeterMasiar yes, we are going to explore the tools suggested there (and the jest textual representations of components concept as well). Thanks! – alecxe Sep 21 '17 at 16:38
  • Have you had a look at the Galen Framework? It follows a different but interesting approach. – beatngu13 Sep 21 '17 at 16:41
  • @beatngu13 oh, wow, yeah, I like it, this may really work, will absolutely give it a try! Thanks again! – alecxe Sep 21 '17 at 16:43
  • 1
    @PeterMasiar With Galen, you use a DSL to write a specification to test against, whereas golden master testing "derives" the expected results from the SUT. I think these are different testing methods, although you can use both for visual testing. Unfortunately, I haven't had the chance to work with Galen, but I'd love to hear how it performs (especially in terms of maintenance effort) compared with tools such as Depicted. So, alecxe, if you have a blog … ;-) – beatngu13 Sep 21 '17 at 17:45

how reliable can it be - would it depend on the browser, resolution and other parameters of the test machine setup?

I can answer this part. I am using Kantu for visual testing, and the tests do not depend on the resolution (the software and its image recognition algorithms account for that). So I can create a test on machine A and move it over to a very different machine B and it still works fine.

The problem with Kantu is that is works only inside Chrome (Chromium). So you can not use it for testing with Edge, Firefox etc.

For cross-browser visual testing we use Sikuli, which runs on the desktop. But while this works with all browsers, it is much more sensitive to resolution changes. So a test created on machine A does often not work on machine B.

  • Oh yeah, I remember we had used sikuli extensively and it was a nightmare to support ever-changing UI, similarity coefficients, handling different systems and browsers. Not sure the cost of maintenance was worth it at the end. Thanks! – alecxe Sep 22 '17 at 14:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.