How to create input data for unit tests for an API client?

Question

I am building the initial set of unit tests for my team's legacy API client system. We have been writing integration tests, but have no unit tests.

It's a Sinatra server that accepts requests from our web app and contacts the third-party APIs. Sometimes it returns the result to the web app.

                                 1                     2
(Web app) ---> Server endpoints ---> API Client class ---> 3rd party API
          <---                  <---                  <---
                                                        3

This being a legacy system, I am thinking about writing characterization tests. I will test the following:

1. mock our API Client class methods, to check how the input data to the server endpoints gets processed and formatted when it reaches our API Client class methods.
2. Similarly mock (or inject) the call to the 3rd party API, to check how the input data to API Client class method gets processed and formatted when it reaches the 3rd party API call.
3. Return canned server responses, to check how our program responds and processes each response.

Since the tests will be characterization tests, I will be just using the output generated by the code run by the test runner.

I am not sure how to generate inputs. The payloads are pretty complex objects, which can have ~30 fields (parameters) for some endpoints.

I don't think it is realistic to test all possible combinations of input parameters. Here's my plan:

• For each test, create a test input that's as typical as possible using the real payload. Save it in a file, as a fixture.
• Identify some fields that are interesting, important, or have previously caused errors. Create test cases for these fields. Load the fixture, and in each test case, overwrite a field in the payload with an extreme value, boundary value, or an illegal value, and making assertions with each payload. For example, if field A should be in range 5 < A < 15, with A = 5, A = 15, A = 4, A = 14, A = nil, etc.
• Run the test with failing expected values. Copy the actual output from the test runner, paste into expected values.

I am not sure if 1. this is the correct way to do characterization tests, and 2. this is a good way to create test input data. Am I overdoing it? Or doing it completely wrong?

Answer 1

You're definitely on the right track here.

I would start by collecting actual requests and responses generated by the live system, then anonymize as needed. These would be the seeds of your tests, since they would have the correct format. They should be available in your application logs, although you may have to do this by using the web application to make the requests yourself.

By collecting a starting set of data from the real system, you'll have something you can modify as you've suggested.

UPDATE (in response to comment from OP):

With the size of the requests (up to 30 parameters), using real data will also allow you to get an idea of the most commonly used parameters and the kind of data real-world usage will give you.

This lets you generate your test data with both the 80/20 (20% of the application gets 80% of the use) rule and risks (the most potentially damaging failures) in mind.

I'd analyze the test data to decide what the most common scenarios are and start with mimicking those as my first priority. My second priority would be the highest risk parts of the system. After that, I'd work my way through a mix of less common scenarios and lower risks until I had coverage that satisfied me.

The important thing to remember with this kind of automation and testing is that you will never be "finished". You have too many potential combinations of data and endpoints to cover all of them. Your goal should be to cover the most common and highest risk scenarios first, then keep adding more until issues that aren't caught by the automation become extremely rare.