2

I would like to start writing some API tests for one of our APIs, but we are seeing a lot of issues with stability of other APIs and I'm not really sure how to handle that problem.

To clarify I have an API A that uses API B,C,D and so forth. And currently I spend a lot of time just figuring out that most bugs are not because of A, but instead of outputs from the other APIs. Preferably I would like to stop doing that type of debugging, because it blocks my own workflow. i.e. I cannot go forward unless my tests are working, and they won't work until another team for the other APIs fixes their bugs.

So I guess the question is: Has anyone experienced a similar situation, and how do you solve it?

A solution I thought of that could work is that I implement a proxy that in some way stores the previous response. It can then during a test check with the previous response to see that our API A is still working and then make a new call to the underlying API and check if they are still working. If they are, then it should store that response, otherwise throw it away and send out a warning. Problem is, that if this doesn't already exist, then it might be expensive to build.

Improve this question

3 Answers 3

Reset to default
1

We have a similar set-up, and your A is what we call a Gateway API. It calls other APIs and might (or might not) add some additional logic.

If the underlying APIs are not your team's responsibility, and you are finding defects in there, then basically you have impediments due to dependencies on a third party (in the broadest sense).

As for storing responses, yes, we do so for all our API tests (currently almost 1000). In a nutshell: we call the API via C# test methods and compare our json against the actual response (using NewtonSoft JSON's DeepEquals method). So we are able to run 1000 tests in a few minutes.

As for your problematic situation, it kind of depends:

  • Have you reported the test failures (impediments) to your lead? What was the response? Will the other teams fix it with priority or not?
  • Depending on the above, for your metrics, would it be useful to execute the test every time resulting in failure? As it is a known defect.
  • Will your team be accounted on the final product quality? I.e. while the underlying APIs are not your responsibility, you might still want to have tests on them to make sure your product at least is fed with quality input.

Edit (to clarify):

If you don't get time to automate API tests, then all you can do is formally present your case one more time and make clear not automating them will cost more in the long run. For example: testing two days manually for every release costs 50 days each year. Automating these cases (set-up included) also costs 50 days, and let's say 10 days a year for maintenance. So over five years you get 250 days versus 100 days. Not to forget, the set-up is the hard part. Once you have a framework, adding tests is easy peasy. So if more features are added, the profit is even larger.

Secondly, to expand on our API testing. We have a framework written in C# to be able to easily send requests and store responses. So, first we manually check an API result (request X should always return response Y) and we store the JSONs used in files. We then write a test method using these JSON files to generate the same request every time, and thus to validate that the response is still exactly the same as when we saved it.

Thirdly, without automated tests you will indeed spend a lot of time, every time again, to test all APIs to see where the error is coming from. The question is, do you need to do a root cause analysis of the issue? And again, make it clear to your manager that you are repetitively losing time on similar issues.

Improve this answer
4
  • I am not sure I understand what you mean by comparing your json against the actual response. is your json a saved one? and the actual response is that from the gateway API or some other API? regarding your points: * we do get prioritised help and the fix the bugs right away, but preferably I do not want to spend time figuring out who is causing the bugs in the first place(me or them). * I am afraid that if we "turn of" a test it will be forgotten. * the problem is just getting the time/resources to write those tests(my manager wont allow it)
    – munHunger
    Mar 13, 2018 at 9:50
  • @munHunger edited answer
    – FDM
    Mar 13, 2018 at 11:52
  • 1
    I think you are absolutely on point! in regards to the question of if I need to do a root cause analysis: yes I do, since someone needs to fix the issue and if it is API Athen I am the one who needs to fix it, otherwise I can just send it down. And I don't want to prematurely send down issues
    – munHunger
    Mar 13, 2018 at 12:27
  • Okay! Good luck!
    – FDM
    Mar 13, 2018 at 12:40
2

I worked on a reasonably similar stack and I would be looking at starting at the ‘lowest’ layer possible and work my way up to testing A.

So, in this case, I would be aiming to test B, C and D. Granted you’re doing some testing of another groups API, but if you’re not responsible for B, C and D, just devise tests to give you sufficient assurance that it worth running your tests against A.

As for checking previous response, tools like SoapUi allow you to store the current response as the expected result and then reuse when running the tests in the future.

Improve this answer
1
  • I am unfortunately 99% sure that I will not be allowed to write tests for another groups API. But using SoapUI sounds good. especially if it is possible to run in a CI workflow
    – munHunger
    Mar 13, 2018 at 7:57
0

Prefereably I would like to stop doing that type of debugging

You are a tester. You shouldn't do any debugging and investigate what is the root cause of the test failure (the only acceptable type of debugging is to reproduce the test flow manually to make sure there is no logical mistakes in your test code). This is the job of dev guys. Moreover you're doing a black-box testing. You should not care of what particular implementation underlies the "box" (API A) you're currently testing.

I cannot go forward unless my tests are working

This is quite a sort of bad practice. You have API. You should have API specification. So having API specs (especially if we're talking about REST api which I presume according to how your question is tagged) makes it pretty easy to write the tests having no working service at all. So just write the tests like the service is working fine and raise bugs once your tests fail in test environment.

Improve this answer
3
  • I should probably clarify that I am the developer of the API. But I do some testing as well, and I am the only one writing automated tests. So you are right, but in this specific case it is important for me to know where the error is, so that I don't have to spend time doing unnecessary work
    – munHunger
    Mar 13, 2018 at 9:34
  • This is not what the integration testing best fits for. You should concentrate on system tests of each particular component's API.
    – Alexey R.
    Mar 13, 2018 at 9:38
  • Re munHunger comment about knowing where the error is... you’re writing tests against api A and you’re the developer of A. As the developer of A, you should be able to capture errors thrown by B, C and D. You could then throw errors back to consuming app. I don’t know how many services calls are made from A to the others, but I can’t imagine it’d take that long to knock up something rudimentary that excercises B, C and D.
    – Chris Adams
    Mar 13, 2018 at 20:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.