4

I am building the initial set of unit tests for my team's legacy API client system. We have been writing integration tests, but have no unit tests.

It's a Sinatra server that accepts requests from our web app and contacts the third-party APIs. Sometimes it returns the result to the web app.

                                 1                     2
(Web app) ---> Server endpoints ---> API Client class ---> 3rd party API
          <---                  <---                  <---
                                                        3

This being a legacy system, I am thinking about writing characterization tests. I will test the following:

  1. mock our API Client class methods, to check how the input data to the server endpoints gets processed and formatted when it reaches our API Client class methods.
  2. Similarly mock (or inject) the call to the 3rd party API, to check how the input data to API Client class method gets processed and formatted when it reaches the 3rd party API call.
  3. Return canned server responses, to check how our program responds and processes each response.

Since the tests will be characterization tests, I will be just using the output generated by the code run by the test runner.

I am not sure how to generate inputs. The payloads are pretty complex objects, which can have ~30 fields (parameters) for some endpoints.

I don't think it is realistic to test all possible combinations of input parameters. Here's my plan:

  • For each test, create a test input that's as typical as possible using the real payload. Save it in a file, as a fixture.
  • Identify some fields that are interesting, important, or have previously caused errors. Create test cases for these fields. Load the fixture, and in each test case, overwrite a field in the payload with an extreme value, boundary value, or an illegal value, and making assertions with each payload. For example, if field A should be in range 5 < A < 15, with A = 5, A = 15, A = 4, A = 14, A = nil, etc.
  • Run the test with failing expected values. Copy the actual output from the test runner, paste into expected values.

I am not sure if 1. this is the correct way to do characterization tests, and 2. this is a good way to create test input data. Am I overdoing it? Or doing it completely wrong?

5

You're definitely on the right track here.

I would start by collecting actual requests and responses generated by the live system, then anonymize as needed. These would be the seeds of your tests, since they would have the correct format. They should be available in your application logs, although you may have to do this by using the web application to make the requests yourself.

By collecting a starting set of data from the real system, you'll have something you can modify as you've suggested.

UPDATE (in response to comment from OP):

With the size of the requests (up to 30 parameters), using real data will also allow you to get an idea of the most commonly used parameters and the kind of data real-world usage will give you.

This lets you generate your test data with both the 80/20 (20% of the application gets 80% of the use) rule and risks (the most potentially damaging failures) in mind.

I'd analyze the test data to decide what the most common scenarios are and start with mimicking those as my first priority. My second priority would be the highest risk parts of the system. After that, I'd work my way through a mix of less common scenarios and lower risks until I had coverage that satisfied me.

The important thing to remember with this kind of automation and testing is that you will never be "finished". You have too many potential combinations of data and endpoints to cover all of them. Your goal should be to cover the most common and highest risk scenarios first, then keep adding more until issues that aren't caught by the automation become extremely rare.

4
  • +1 A common situation and Kate provides a great answer. Use info from real requests! This may mean approaches such as using chrome tools and looking at network requests in them to see exactly what requests and params are used, or, as Kate mentions use logs as available. – Michael Durrant Aug 31 '20 at 14:31
  • Thanks! Another (more important) question: I need to test different combinations of inputs, but because there can be up to 30 different parameters to an endpoint, there can be a giant number of combinations. Testing every combination sounds impossible. Should I rely on sampling? Also, is it okay to just test one parameter value at a time? – user25409 Sep 1 '20 at 0:20
  • @user25409 - Real data will help you with this as well: it will give you an indication of the most common data combinations. It's perfectly fine to test changing one parameter value at a time. – Kate Paulk Sep 1 '20 at 11:17
  • @KatePaulk Thank you! – user25409 Sep 1 '20 at 18:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.