9

I've recently stumbled upon this new (at least to me) testing type: Snapshot Testing:

For Facebook's native apps we use a system called “snapshot testing”: a snapshot test system that renders UI components, takes a screenshot and subsequently compares a recorded screenshot with changes made by an engineer. If the screenshots don't match, there are two possibilities: either the change is unexpected or the screenshot can be updated to the new version of the UI component.

Has anyone tried that? When is this type of testing applicable? Is this a replacement for unit testing or a just a way/style of writing them? Is it only applicable for the UI tests?

  • 1
    Maybe it's a testing of snapshot version of an artifact in comparison to release version? Looking into your link I would say I'm wrong. But links tend to break, so perhaps it would be better to quote part of the link content here? – dzieciou Sep 20 '17 at 13:31
  • @dzieciou good idea, replaced the link with something more generic and added a quote. Thanks. – alecxe Sep 20 '17 at 13:50
5

As already pointed out by Peter Masiar, you basically compare against a "gold standard". Therefore, I would say that this is a derivative of characterization testing/golden master testing:

Characterization Testing (aka Golden Master Testing) is a means to characterize the behavior of the test object to protect it against unintended changes, regardless of its correctness. In order to do so, the results of a previous (typically stable) version—the golden master—serve as the oracle for the tests (also known as consistency oracle since it compares the consistency between two versions).

Characterization testing (coined by Michael Feathers in his book "Working Effectively with Legacy Code") comes in many different flavors:

Some do pixel comparison, others do text comparison, and there are also hybrid approaches (mentioned by roesslerj). Each variant has its pros and cons, but essentially it is the same mindset: compare against a previous version, regardless of its correctness. There is also a very cool introduction ("the dancing pony" starts at 4:26) to the basic concept by Google, who develops Depicted.

Going back to Jest, the quote you have in your question might be a bit misleading because in the case of Jest "Snapshot" does not mean "Screenshot", i.e. there is no pixel comparison. Consider the following example from their docs:

import React from 'react';
import Link from '../Link.react';
import renderer from 'react-test-renderer';

it('renders correctly', () => {
  const tree = renderer.create(
    <Link page="http://www.facebook.com">Facebook</Link>
  ).toJSON();
  expect(tree).toMatchSnapshot();
});

What Jest does is it takes a React tree and serializes the HTML/JavaScript it generates to a so-called "snapshot file" (.snap suffix):

exports[`renders correctly 1`] = `
<a
  className="normal"
  href="http://www.facebook.com"
  onMouseEnter={[Function]}
  onMouseLeave={[Function]}
>
  Facebook
</a>
`;

This is now your golden master, which contains a textual representation of the rendered React component:

On subsequent test runs Jest will simply compare the rendered output with the previous snapshot. If they match, the test will pass. If they don't match, either the test runner found a bug in your code that should be fixed, or the implementation has changed and the snapshot needs to be updated.

If the tested React tree changes (e.g. to a different link), you will get a simple textual diff:

Failed snapshot test.

According to the Jest team, this has the following advantages over pixel comparison:

  • No flakiness: Because tests are run in a command line runner instead of a real browser or on a real phone, the test runner doesn't have to wait for builds, spawn browsers, load a page and drive the UI to get a component into the expected state which tends to be flaky and the test results become noisy.
  • Fast iteration speed: Engineers want to get results in less than a second rather than waiting for minutes or even hours. If tests don't run quickly like in most end-to-end frameworks, engineers don't run them at all or don't bother writing them in the first place.
  • Debugging: It's easy to step into the code of an integration test in JS instead of trying to recreate the screenshot test scenario and debugging what happened in the visual diff.

On a final note, Jest is not exclusive to React. It has the concept of serializers, which take care of reading/writing snapshot files. There are default serializers for built-in JavaScript types, HTML elements, ImmutableJS, and for React, but you can also write your own serializer.

So, you could theoretically write unit, integration, and system tests with Jest. I just don't think that it always makes sense. For instance, when you have a simple add function that sums up two integers, why would you use a snapshot file for the result? A plain old assertion does the job.

  • 2
    Excellent, upvoted. In my experience, trying to bypass the flakiness by using different tool (not a real browser) would likely cause different kind of flakiness: caused by subtle differences in the implementation of DOM processing between that tool and real browser. It is "pick your poison" game, with no real good answers, there is no silver bullet, AFAIK. I would suggest "less flakiness". "No flakiness" is a promise of snake oil salesman and not your own words - likely you just copy-pasted from the marketing page. :-) Is Jest open source? – Peter M. - stands for Monica Sep 21 '17 at 15:00
  • 2
    @PeterMasiar Totally agree. Every time I read "no flakiness", I just think "naaah". But I work for a tool vendor myself, so who am I to say something … ;-) – beatngu13 Sep 21 '17 at 15:12
  • We need here insight not only from the guys in the trenches (like mine), but also insight from the guys who are so to speak "are making our weapons" - great to have you here! – Peter M. - stands for Monica Sep 21 '17 at 16:28
  • See also comments by @beatngu13 about Galen in sqa.stackexchange.com/questions/29735/… - – Peter M. - stands for Monica Sep 21 '17 at 18:40
7

This kind of testing is testing only a visual appearance of the web pages, so it is in no way a valid replacement for unit tests (which test internal logic of the modules). But it might be a part of integration/system tests.

Applitools.com calls it "visual testing" and provide framework for it. Their framework allows to ignore parts of the screen image (say the part where date/time is displayed), compares the rest pixel-to-pixel, and highlights the differences with "gold standard", if any. With a click you can accept new screen as new "gold standard".

Such visual tests cannot replace standard webdriver-based integration/e2e tests, because it is limited to images only. So for instance cannot perform activities which selenium can, like: pick a random a "thingie" from a list on a page, go to different page, and check if thingie is present (or not present) as required by business logic. For that, it would require powerful OCR, and much more programming (and it would be waste of time, because selenium/webdriver can already do it, and much easier).

But such visual testing is excellent complement to selenium/webdriver testing, because it checks exactly the aspects of the page which selenium ignores. So it is excellent additional layer of defense, after unit tests for libraries, unit tests for single page applications (Angular/protractor), and e2e selenium tests (in the order of preference).

Disclosure: I don't work for applitools and don't use their tool, but I looked a demo and the technology looked cool - it is on my list to try sometimes in the future.

  • 2
    Good answer, but one should note that in the case of Jest "Snapshot" doesn't mean "Screenshot", i.e. there's no pixel comparison. See their FAQs: "With Snapshot testing values are serialized, stored within text files and compared using a diff algorithm." – beatngu13 Sep 20 '17 at 21:32
  • @beatngu13 - I am not sure how Applitools works (from the short intro it looks like image processing), and I am also not sure how diff would be useful if it is NOT image processing (pixel-to-pixel compare). What is "Jest Snapshot"? Care to write answer explaining the difference, with links if you have any? – Peter M. - stands for Monica Sep 20 '17 at 21:39
  • 1
    Here's my answer, hope it helps. – beatngu13 Sep 21 '17 at 8:01
  • @beatngu13 - excellent, upvoted. – Peter M. - stands for Monica Sep 21 '17 at 14:53
3

There is also a hybrid approach to that, which is implemented by tools like ApprovalTests, TextTest and ReTest. The general idea is instead of checking individual elements, checking everything at once. If the contents hasn't changed, then the layout should not have changed as well.

The problem with pixel-compare tools is bulk-changes. If you change the Logo, each and every test fails and has to manually be reviewed / approved.

Disclosure: I work for ReTest.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.