Equal or different test data for higher test levels

Question

It's just a little issue, but one that pops up in my mind regularly when writing test scripts.

Imagine you have an application which does some car tax calculation, having a REST service which has a method for calculating the tax and an HTML interface with a screen to enter the required input and display the tax to pay. The HTML interface makes uses of the REST service and does nothing more than some input validation, passing the values to the REST interface and then display the tax to pay.

Let's assume you developed several service tests for the REST service, e.g. using SoapUI, some with invalid input (weight=0kg or price=€0) and one with valid input (weight=900kg, price=€25000).

The HTML interface is tested for usability, browser compatibility and other relevant quality attributes, which do not apply for the service test. The system test, e.g. using Protractor, contains some test cases concerning input validation, like an empty field for the weight of the car. You also want to verify if the tax calculation displays the correct amount to pay for valid input. Would you use the same test data as used in the service test, because you have the expected results already available, or would you deliberately choose different test data in order to implicitly test the business logic more thoroughly?

Let's assume two situations:

1. You know the unit tests concerning the tax calculation have a coverage of about 80% and you know the developers are aware of techniques like equivalence partitioning. A decent base for the test pyramid.

2. The unit tests concerning the tax calculation have a coverage of just 40% and there is no time and budget to write more unit tests and raise the coverage, unfortunately, a common situation.

In the latter situation, you could, of course, extend the service tests to increase coverage on a higher level, but even then you have to decide if you want to deliberately use different input values for the system test.

Answer 1

Personally, as the "service test" is the end-to-end test and the expensive one, I'd just put in some simple values that I know are good. As this is a single test with fixed parameters, I'd just pick some numbers. I'd likely pick different numbers than in lower level tests if they had fixed values, just for the slight increase in coverage. (Though hopefully the lower level tests could have some properties based testing so these values would matter much less.)

For me, end-to-end tests tend to just be happy path verifications. If there's known regression cases I want to handle, I'd have tests for those too, but if possible, I'd be pushing them down lower in the pyramid (e.g. if I know it's something the API should handle, I'd write an API test, and not an end-to-end test).

The idea of using end-to-end tests to increase coverage that should have been done lower down the test pyramid just screams broken to me. I'd want to add the tests closer to the bottom of the pyramid - building out the breadth at the top of the test pyramid is almost always wrong.

Answer 2

Would you use the same test data as used in the service test, because you have the expected results already available, or would you deliberately choose different test data in order to implicitly test the business logic more thoroughly

The short answer: No

The longer version: If the business logic is not being tested at the unit test level, then I'd look at writing more unit tests, not adding testing at the more expensive integration/API and UI levels.

There are a few exceptions to this rule, but for the most part, you don't want to build automated tests at a higher level than you have to. Business logic should be unit tested because it has no impact on the UI or the API (unless they are legacy code or badly built with the logic and presentation intertwined in which case you get the main exception below).

The biggest exception to this would be a situation where it was necessary to build automated testing for something that was not unit-testable (I have been in this situation far too often). At one point I was supporting an application which was originally written in Pascal, ported to Delphi, and most of the legacy intertwined business logic and display rules were still used in the code, making it impossible to unit test. The lightweight tax calculation regression test suite took approximately 24 to 36 hours run-time (it was split between several machines to allow it to be run daily). The same set of tests run as unit tests would have taken less than an hour and been much less fragile.

The other major exception is a situation where your application is consuming an API or other service maintained by a third party. You can't unit test the other service, and your testing is effectively limited to pushing data in, checking that the data that comes back is what it should be, and checking that the output data is properly displayed.

If your developers don't have the time to build out the unit tests to cover the business logic effectively, your best option if you have the programming skills and IDE licensing available is to get permission for your team to write them. You'll get a lot more testing ROI out of that than building automation further up the chain.