1

Here's the situation: I have a number of APIs that are unit tested. A swagger spec (swagger 2.0) is auto generated from code annotation on the APIs.

I'm tasked with validating the API endpoints. In order to do this, I import the spec into Postman and generate a request collection to work with.

I'm currently considering validating the JSON responses against a schema in automated test scripts, but it this really useful? Because the collection I'm working with was generated based on swagger spec that in its turn was generated from the code.

1
  • Excellent question!
    – dzieciou
    Mar 6 at 18:38
1

Postman collection generation from API just validates the API definition like:

  1. Resource (End point)
  2. HTTP Method
  3. Parameters

It doesn't validate actual response.

imagine that the url of the generated collection was:

{{baseUrl}}/user?id=54750966

change baseUrl variable value as :

https://reqres.in/api/users?page=2 #

Now goto API and click validate , you can see that the API validation passes even tho we have entirely different baseURL.

Also validation is not real time , unless you go and explicitly validate collection against api definition we won't detect changes.

Summary:

  1. Use collection generation for initial setup
  2. Make sure you validate responses as generated collection doesn't do this for you
4
0

I think you're missing you're also responsible for figuring out whether that Swagger documentation makes sense. It can be broken in various ways:

  • missing (information about) authentication
  • missing status codes that the endpoint in reality return
  • query parameters marked as path parameters or vice versa
  • spelling mistakes
  • example request and response bodies that actually don't work or that are not consistent across endpoint

By coincidence I'm currently busy with testing APIs as well and at different times, I stumbled upon every one of the points above. Plus, a few times, the Swagger documentation was so wrong it couldn't even be generated properly, and so any paste and copy of the link into Postman proved useless.

To me, doing validation is not enough however I look at it. If you're tasked with this tiny subset of what you can do, it's still not enough to copy and paste the Swagger link into Postman and auto-generate a collection out of it. The difference between reality and Swagger could be bigger than what's acceptable. It's more likely that you'll need to figure out bits and pieces here and there and manually update the collection that was generated out of the Swagger.

Because the collection I'm working with was generated based on swagger spec that in its turn was generated from the code.

Auto-generation doesn't mean it was generated correctly. In suprisingly many instances, it's not because ... well I suppose the code could be a mess in the first place, so then the Swagger doesn't make sense either. So if you take it as a source of truth, you're trapped from the very beginning.

1
  • Swagger as documentation is only used internally in the company I work for, and doesn't get that much prio because the user pool for that documentation is dozens instead of thousands. So even if I would raise some bug tickets for issues I found, it wouldn't get fixed. Besides that, you really have some valid points though, but a bit off topic for the question I'm asking.
    – PixelPlex
    Mar 8 at 8:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.