1

I started using a nice NuGet package for doing integration tests - https://github.com/VerifyTests/Verify and received feedback such as :

it's not entirely fair to use the system under test response as expected instead of creating it manually.

In other words, I would be expected to

I want to mention here that we have multi-country scenarios with quite a few country specific flags, which is why this tool helps me have country agnostic tests in terms of code (and only the "expected" is country specific)

Basically this tool helps that after a test is written, on the first run it fails and suggests what the expected result should be - it assumes that what I am asserting is what I want as expected (which I was told is a bad practice to let the tested application "decide" what the expected should be).

In my tested application I deal with huge json files with lots of computed properties, which takes a lot of time to create from scratch.

On the first run of the test we get something like this, where on the left is what the system under test result is found and on the right we have to put the expected value (if the received result is correct, copy paste) enter image description here

If the application result is correct (I validate every single property) and then I simply paste the received result in the "verified" file and from there on, the test will forever pass until the received values do not match the expected value.

Question: Is this type of snapshot testing a good practice or not?

Changed question as to be more on the point.

Question: Does this type of snapshot testing break any testing rules?

(Provided that when defining the test and getting the initial result from the application, we as testers actually check that result before making it the expected result)

Improve this question

1 Answer 1

Reset to default
3

Is this type of snapshot testing a good practice or not?

It depends. The value of any practice depends on its context (one of the Context-Driven Testing School principals).

The snapshot tool will help exactly as they say in their README:

The test will fail if the two snapshots do not match: either the change is unexpected, or the reference snapshot needs to be updated to the new result.

If it helps your testing activities, then probably is a good practice. You must also analyze alternative practices, maintenance/usage cost, your people, and your resources. E.g., if the results change often, but you are interested in only a few attributes, or if you don't have the budget to execute the snapshots checks, etc. Maybe there are better practices to fulfill your testing goal(s).

received feedback that it's not entirely fair what I do

What is the rationale here?

Improve this answer
2
  • Added context regarding the feedback received: "it's not entirely fair to use the system under test response as expected instead of creating it manually". The issue seems to mostly be because it's somewhat automatically generated (rely on the tested application for first expected result, although it is validated, not just copy pasted).
    – Claudiu A
    Jan 27, 2022 at 19:59
  • 1
    "it's not entirely fair" the use of fair here seems strange, as if they would expect to certain level of effort to put, rather than finding ways to make things easier. "rely on the tested application for first expected result" You are using a tool to support your testing - if your testing goal is to investigate the changes since the last snapshot, then the tool can be useful.
    – João Farias
    Jan 31, 2022 at 10:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.