Are there common techniques for testing the conformance of an implementation to a general contract?

Question

Say you have defined some abstract interface and you specify a general contract for that interface to which all implementations must adhere. Are there common techniques that can facilitate testing the conformance of arbitrary implementations to that general contract?

In my environment we have a large collection of such abstract interfaces that define a common interface to several different hardware systems. My current approach is to define a set of conformance tests that work solely with the abstract interfaces. These tests are then packed into a library which again is imported by the actual unit tests of the implementations. These instantiate the tests and also provide a factory which generates concrete testee instances that will be tested by the conformance tests.

In theory this approach sounds fine, but in practice writing the conformance tests is very difficult and the value of their results appears to be very limited, because a lot of additional information about the testee instances is required.

Example (pseudo code):
the abstract interface is

myinterface
{
// Triggers some behaviour and returns immediately. While the behaviour is active,
// getState() returns *Running*. If an error occurrs the state changes to *Halted*.
function run();
function getState : State;
};

This interface and its contract are very simple, yet the test case for this is (relatively) difficult to write and its result says little about whether or not the implementation is doing the right thing or not. Is my approach reasonable or would I be better off just writing separate unit tests for all implementations? Are there other techniques that could help to achieve what I want (i.e.: guarantee the conformance of different implementations to the contract of their general interface).

Answer 1

Yes. I call them Contract Tests. One easy way to understand them:

Start with tests for a specific implementation of the interface. For example, consider ArrayList implements List. You write tests for ArrayList. One test could be this:

testEmptyIffZeroSize:
  list = new ArrayList()
  assert list.isEmpty()
  assert_equals 0, list.size()

Notice that LinkedList should also pass this test, but there's only one difference:

list = new LinkedList()

So imagine you have ArrayListTest and LinkedListTest that both have this test, but the only difference is the class name used to instantiate list. Of course, in both cases, list is just a List. You could make these tests identical by extracting a method createEmptyList() : List in each test class. Now you have this:

class ArrayListTest
  List createEmptyList():
    return new ArrayList()

  testEmptyIffSizeZero():
    List list = createEmptyList()
    assert list.isEmpty()
    assert_equals 0, list.size()

class LinkedListTest
  List createEmptyList():
    return new LinkedList()

  testEmptyIffSizeZero():
    List list = createEmptyList()
    assert list.isEmpty()
    assert_equals 0, list.size()

This time I showed the type of list to emphasise that I've made the tests identical.

Now, remove duplication by introducing a new superclass, pulling up the test, and declaring createEmptyList() as abstract.

abstract class ListContract
  abstract List createEmptyList()

  testEmptyIffSizeZero():
    List list = createEmptyList()
    assert list.isEmpty()
    assert_equals 0, list.size()

That is a Contract Test. It should pass for all implementations of List. When you implement List, subclass ListContract and implement createEmptyList().

Now, you can either extract contract tests from existing implementation tests, or you can draft contract tests as you design interfaces. I do both.

I hope this helps.

UPDATE 2020-04-29: If you don't like to subclass the contract tests in order to write tests for each implementation of List, then combine the contract tests with an Abstract Factory that creates instances of the various implementations of List. This would move createEmptyList() onto the Abstract Factory and you'd implement the Factory for each implementation of List. This uses composition instead of inheritance in the contract tests.