I am testing few API endpoints of a product and there are many observations like:

  1. Server response with 500 error if the application content type is JSON instead of form data.
  2. Different endpoints provides different error response for the same invalid body. For instance, if the file is not provided then one endpoint response as [ error: file is not provided ] and another as {File: "This is a required Field}
  3. Endpoints that point to different versions of a product return response with different character case, eg: in version 1 its {File:"something}, v2 {file:"something"} and v3 its {FILE:"something"}

I raised these as bugs but the developer tells no one will do these things, as everything is well written in the API contract

Does these counts as valid observations? and if not then why would we text negative test cases, as the proper use case is already mentioned in the contract and lets blindly believe that users will follow it.

  • Why would someone down vote this ? – PDHide Feb 25 '20 at 20:19
  • Please comment something before downvoting – PDHide Feb 25 '20 at 20:30

Does these counts as valid observations?

Without going into details then yes, those are valid observations and concerns that should be raised, and probably tracked as issues.

But that doesn't mean those concerns should be fixed now or ever, your testing does not live in a vacuum and should take into account user expectations and their legacy code, the project's deadlines the effort the fix the problems and most important the harm if a user stumbles upon one of those issues.

I use the term issue since those are not necessarily bugs in the context of your project. You want to document your findings for future reference or improvement work, but you could use tasks, feature requests or something similar that will not increase bug count.

If you are worried then raise the question in a wider forum or to other managers, but be prepared to accept the same answer.

why would we text negative test cases, as the proper use case is already mentioned in the contract and lets blindly believe that users will follow it.

There is legacy behind this code, there is probably an obscure reason why it was built like that. You should continue testing trying to find real and important problems, while raising concerns on the way. Those concerns might add up building a "weight of the evidence" that eventually will lead to some action.

Project managers are fine with a casual low severity problem here and there, but if they find out over time that there's enough problems accumulated they might take action to fix them.

  • This makes sense, thank you for the comment. would you suggest fixing it for a new project with fair deadlines ? – PDHide Feb 25 '20 at 15:28
  • 1
    A lessons learned from previous projects is always a good start for a new one, those problems could,and should be, avoided instead of fixed. – Rsf Feb 25 '20 at 15:57

Going through your particular points I would say that

  1. This is the valid observation since this is improperly configured request (aka client error) so it has to be responded with 4xx status code. In your case there is obviously lack of request validation on the server so that it tries to execute business logic against incorrect data which might result in different issues starting from performance degradation finishing with the entire service halting.
  2. This is arguable point since this is a human readable (not for automated processing or so) message and in general both messages describe the issues in the well understandable manner
  3. I would say this is also a valid observation since when you de-serialize the object you might go into issue when the client classes wouldn't match the case of their fields with the ones received from the server.
  • 1
    thank you for the reply , but the case sensitive part is not about the header, but the response . – PDHide Feb 25 '20 at 15:22
  • Oh,, are these JSON objects sent within response bodies? – Alexey R. Feb 25 '20 at 15:23
  • stackoverflow.com/questions/2738321/… , from this question we can see that JSOn is indeed case sensitive (and unless the server handles it as case insensitive) – PDHide Feb 25 '20 at 15:23
  • yes , they are the response body i get back from different versions of the endpoint – PDHide Feb 25 '20 at 15:24

Well, I think you're focusing too much on just bugs, but they do not exist in isolation. What is your situation? What is your context? E.g. case-sensitiveness might be important if the error goes all the way to a frontend or when it's read by another software that works case-sensitively. You need to know your context, so you know what bugs are important bugs you need to focus on, and what priority/severity assign to your bugs.

One exception among your example might be the first case with response code 500. In my opinion, API should never return this, it should return a more suitable error code, in this case perhaps 415 or 400.

I'd suggest talking to other members involved in developing that software, they might provide you with more context, then you can decide whether or not to insist on these issues.

  • "case-sensitiveness might be important if the error goes all the way to a frontend or when it's read by another software that works case-sensitively.." when testing from a component test level, Does the component need to assume that the consumer of the api service will implement case-insensitivity , or develop a api service that works in both the cases – PDHide Feb 25 '20 at 16:02
  • Also, about the status code, even that's out of context. The consumer wouldn't be sending a non-json data or non-form data if the API contract clearly states that's what is expected. Thank you for the comment again, please do reply your thoughts – PDHide Feb 25 '20 at 16:16

I do think your findings are worth discussing with the development team, they could be defects. The push back of the developers is understandable, in this case they do not see the risk, certainly not one that is worth doing rework.

This is a common pattern I see in teams where testers are not collaborating with developers before they start coding. Breaking the system after it was build gives insights, but I would rather focus on building the best system together from the start. Becarefull that you do not swamp a development team with trivial defects. Defects that not really improve the value of the product.

The JSON example is a risk, probably the behaviour could have been discussed earlier in the development process. During a Three-Amigo session try to challenge with "But what if we pass the API invalid form-data? for example a JSON". The developer could suggest a 500 error. Now you can discuss this by protesting and saying our API guidelines say we should return an object with a clear error message.

I like API consistancy, but if your consumers are only your own developers, they might not think the refactor is worth their time. Maybe you lack API guidelines, how you can improve over time?

How can you start to have the right discussions at the right moment. A conversation about defects/risks with just testers and developers is a risk in itself. What is the business value? Involve business minded people. How much time should we spend on technical-debt? Will we go faster or slower?

  • I agree with that and j am an advocate for pushing QA to the left , making them involved as early as requirement gathering. But that's not the culture in the organization and sometimes it's too hard to be a quality assurance engineer in such a workplave – PDHide Feb 27 '20 at 12:25
  • 1
    You have to teach them and change the culture. In most larger companies this is not the culture. Change your organization or change your organization ;-) – Niels van Reijmersdal Feb 27 '20 at 12:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.