What are the scopes of API testing

Question

I am testing few API endpoints of a product and there are many observations like:

1. Server response with 500 error if the application content type is JSON instead of form data.
2. Different endpoints provides different error response for the same invalid body. For instance, if the file is not provided then one endpoint response as [ error: file is not provided ] and another as {File: "This is a required Field}
3. Endpoints that point to different versions of a product return response with different character case, eg: in version 1 its {File:"something}, v2 {file:"something"} and v3 its {FILE:"something"}

I raised these as bugs but the developer tells no one will do these things, as everything is well written in the API contract

Does these counts as valid observations? and if not then why would we text negative test cases, as the proper use case is already mentioned in the contract and lets blindly believe that users will follow it.

Answer 1

Does these counts as valid observations?

Without going into details then yes, those are valid observations and concerns that should be raised, and probably tracked as issues.

But that doesn't mean those concerns should be fixed now or ever, your testing does not live in a vacuum and should take into account user expectations and their legacy code, the project's deadlines the effort the fix the problems and most important the harm if a user stumbles upon one of those issues.

I use the term issue since those are not necessarily bugs in the context of your project. You want to document your findings for future reference or improvement work, but you could use tasks, feature requests or something similar that will not increase bug count.

If you are worried then raise the question in a wider forum or to other managers, but be prepared to accept the same answer.

why would we text negative test cases, as the proper use case is already mentioned in the contract and lets blindly believe that users will follow it.

There is legacy behind this code, there is probably an obscure reason why it was built like that. You should continue testing trying to find real and important problems, while raising concerns on the way. Those concerns might add up building a "weight of the evidence" that eventually will lead to some action.

Project managers are fine with a casual low severity problem here and there, but if they find out over time that there's enough problems accumulated they might take action to fix them.

Answer 2

Going through your particular points I would say that

1. This is the valid observation since this is improperly configured request (aka client error) so it has to be responded with 4xx status code. In your case there is obviously lack of request validation on the server so that it tries to execute business logic against incorrect data which might result in different issues starting from performance degradation finishing with the entire service halting.
2. This is arguable point since this is a human readable (not for automated processing or so) message and in general both messages describe the issues in the well understandable manner
3. I would say this is also a valid observation since when you de-serialize the object you might go into issue when the client classes wouldn't match the case of their fields with the ones received from the server.

Answer 3

Well, I think you're focusing too much on just bugs, but they do not exist in isolation. What is your situation? What is your context? E.g. case-sensitiveness might be important if the error goes all the way to a frontend or when it's read by another software that works case-sensitively. You need to know your context, so you know what bugs are important bugs you need to focus on, and what priority/severity assign to your bugs.

One exception among your example might be the first case with response code 500. In my opinion, API should never return this, it should return a more suitable error code, in this case perhaps 415 or 400.

I'd suggest talking to other members involved in developing that software, they might provide you with more context, then you can decide whether or not to insist on these issues.

Answer 4

I do think your findings are worth discussing with the development team, they could be defects. The push back of the developers is understandable, in this case they do not see the risk, certainly not one that is worth doing rework.

This is a common pattern I see in teams where testers are not collaborating with developers before they start coding. Breaking the system after it was build gives insights, but I would rather focus on building the best system together from the start. Becarefull that you do not swamp a development team with trivial defects. Defects that not really improve the value of the product.

The JSON example is a risk, probably the behaviour could have been discussed earlier in the development process. During a Three-Amigo session try to challenge with "But what if we pass the API invalid form-data? for example a JSON". The developer could suggest a 500 error. Now you can discuss this by protesting and saying our API guidelines say we should return an object with a clear error message.

I like API consistancy, but if your consumers are only your own developers, they might not think the refactor is worth their time. Maybe you lack API guidelines, how you can improve over time?

How can you start to have the right discussions at the right moment. A conversation about defects/risks with just testers and developers is a risk in itself. What is the business value? Involve business minded people. How much time should we spend on technical-debt? Will we go faster or slower?

Answer 5

If the API is properly documented, the test should be making sure that the API server actually behaves as documented in the specification. What proper documentation is depends on the type of API. If it's a REST API, documentation should be in the OpenAPI format, and if it's a GraphQL API, it should an SDL schema.

If you have proper documentation, you can also use frameworks like Dredd and schemathesis, which can run automatic validation tests against the API from the specification. schemathesis is also easily extendable with custom test cases.

Your points are perfectly valid from the API design point of view. A 500 should preferably return a JSON payload if the other responses also come in JSON, consistency across error messages is also better practice, and consistency in the shape of attributes across different versions of the API helps with usability. However, if these behaviours are correctly documented, it shouldn't be a problem for the API consumer. If you have an API client development team, I'd liaise with them to see if the find the API sufficiently usable in its current state.