What is really tested in an integration test?

Question

what is really the difference between a component/unit test and an integration test?

Suppose you do very fine unit testing with stubs and drivers and test each component as far as you can with unit tests, which kind of tests would you then do in integration testing?

Things I can think of:

(looking bottom up, from unit to integration level)

• If you test on unit level with BVA, perhaps it's sufficient to only
  test with EP on integration level?
• You can do positive and negative testcases on unit level, while you would restrict yourself to more positive testcases on integration level?
• On integration level you focus more on data. But how would you do this exactly?

(looking top down from system to integration level)

• You see which use cases can be sensibly executed with the few components that are integrated already.

So if you do not want to do double work, which NEW tests would you write for integration level?

Answer 1

Welcome to SQA, Chris.

First, regarding terminology, there are many terms for describing different kinds of testing. Everyone uses them differently. In some situations, those terms have specific meanings defined in contracts or regulatory documents. Usually though, the terms are just labels assigned to vague concepts that individuals (rather they realize it or not) do not necessarily agree on. When someone talks about "unit testing" or "integration testing", I try to ask what those terms mean to them so that we speak the same language. In any case, I will assume your question does not apply to a project with specific test processes defined in a contract or a regulatory document.

Yes, if you unit test with boundary value analysis (BVA), equivalence partitioning (EP) should suffice in higher-level tests.

I do not mean to be evasive, but your integration tests should be whatever they need to be. I think unit-testing is about testing individual (and perhaps indivisible) components, and system testing is about ensuring that the system works as a whole. Software is an assembly of components, but a component may itself be a sub-assembly of other components. Unit testing only verifies that a component behaves as the developer intended. It does not guarantee that a component behaves the way everyone else assumes it behaves. Integration testing is an opportunity to verify those assumptions.

It may help to think about the system-test/integration-test/unit-test hierarchy as the inverse of the diagnostic process. When the dishwasher repairman arrived to fix my dishwasher, he started diagnosing the problem by interacting with the system interface, i.e. he closed the door, pressed some buttons on the control panel, listened to the sounds coming from the dishwasher, and then opened the dishwasher back up to see whether the dishes were wet. After that, he narrowed his search to a specific sub-assembly, which has its own interface that is different from the system's. Eventually he narrowed down the problem to a specific part, which has a different interface from the system's or the sub-assembly's.

Integration testing is a means of identifying problems in a more easily-diagnosable manner than system testing. The problems are more easily diagnosable because the integration test is specific to a sub-assembly (with fewer moving parts) rather than to the system as a whole. Because it works at the sub-assembly level, the integration test may have access to addition diagnostic information that is not exposed at the system level.

Looking at this from the other direction, then, your integration test should focus on the sub-assembly's interface, not the interfaces of its component parts. If you find a large overlap between your component tests and your sub-assembly tests, it may mean that you do not really need to test the sub-assembly, or conversely that you need to define your sub-assembly in a different way.

Answer 2

The objective is to resolve conflicts between components being integrated. Those kind of conflicts cannot be found during unit tests. The example of dishwasher of @user246 is very illustrative here.

Type of conflicts you may encounter:

• A component transmits syntactically wrong or no data. The receiving component cannot operate or crashes (functional fault in a component, incompatible interface formats, protocols faults).
• The communication works but the involved components interpret the received data in a different way (functional fault of a component, contradicting or misinterpreted specifications).
• Data is transmitted correctly but it is transmitted at the wrong time, or is late (time problem), or intervals between the transmissions are too short (throughput, load, or capacity problem).

Based on the "Software Testing Foundations" book by Spillner et al.

Note, you may find those problems both during integration of your software and of your software with off-the-shelf components, e.g., libraries.

EDIT: I started to analyze different bugs that I and my colleagues found in integration tests. I believe it can be useful to know also examples of the conflict classes I mentioned above. Being more sensitive on what can go wrong in integration can allow us to stress those points in tests.

EDIT 2: I found a more comprehensive source on integration bugs found "Empirical Study of Software Interface Faults — An Update". It's from 1987 but does not seem much outdated.

Answer 3

I agree with user246 in regards to always confirming implied meaning of terminology. I was taught and have always used this reference wrt understanding the abstract "levels of testing" and wrote a bit about it here.

In my current role I manage a team of testers that do 90% integration testing. What that means is that while we rely on our developers to do unit testing of individual functions and components prior to any check-in. Oour responsibility lies primarily in testing functional "scenarios" of the 'components' below or without the user interface. This is beneficial because often time the functional capabilities or "business logic" is well tested before the user interface stabalizes enough to automate tests. (Also IMHO, it is more effective to automate "functional" tests that exercise the "business logic" below the UI, and use UI automation to mostly focus on behavioral aspects of the UI code.)

Also, as Sam indicates above unit tests may rely on stubs or mocks, integration tests should never use a mocked component.

WRT BVA and EP, I do think that boundaries should be throughly tested in unit tests. But, EP is a mechanism that can help identify sub-boundary conditions at the edges of 'ranges' of data partitions in some cases. The edge cases of EP (e.g. unique or special sets of EP data) can typically be tested more efficiently at the integration level. (Efficient in terms of time. Unit tests can't take forever, or be completely inclusive. In our case our integration tests are developed in-sync with the dev code in an effort to identify "functional" issues way up stream before they are incorporated into the main branch.