We currently have a rather rich subset of end-to-end tests for our AngularJS applications. And, we do get a lot of visual/layout/design issues on every single development iteration. Usually, we do not cover this kind of issues with end-to-end tests and testing the fixes manually. The reasoning was always - too hard, too fragile, too expensive to automate.
In some cases though, a style of an element or block on a page is really critical for an end-user and it would make sense to have an automated end-to-end test checking it. We try to check the colors, font styles and other CSS properties inside our Protractor tests, but it is still checking only these specific things and not an element or part of a page as a whole. Plus, the assertions themselves are not readable and easy to break, for example:
expect(somePage.someElement.getCssValue("background-color")).toEqual("rgba(255, 211, 205, 1)");
We are thinking of adding some sort of a visual component to our tests. Ideally, comparing screenshots of what we expect an element to look like and how it actually looks. But, there are a lot of open questions at this point:
- how reliable can it be - would it depend on the browser, resolution and other parameters of the test machine setup?
- is it possible to integrate with our existing Protractor test codebase or we need a separate specialized tool for that?
- test result reporting - how would we present the test results?
Have you done something similar and how did you address the above mentioned issues? Was it worth the cost of maintenance?