One suite of tests for different implentations of a protocol

Question

We have a logging protocol that defines how output of logging libraries should look like, e.g.:

• each line should be a JSON
• each line should not exceed 1MB

There are multiple implementation of this logging protocol, i.e, logging libraries written in different languages (Java, Python, etc.). I would like to create one set of tests that I can run against any logging library to verify it complies to the logging protocol. I don't want authors of a new library to port existing test. Instead, I want them to reuse existing tests.

The are already known solutions to that like Reactive Streams Test Compatibility Kit, but they work for implementations in the same language (or at least for JVM-languages). In my case implementations can be of different languages.

How can I make tests language independent?

Initial idea

1. My first idea is based on assumption that logging libraries always take one or two arguments:

   logging.error("Serious problem occurred", exception)
   

   and log output to a log file

   { level: "error", message: "Serious problem occurred", stacktrace: "....."}
   

2. The test harness would pass some string to the testing library and verify the output.

3. The test harness would communicate with a logging library through a pipeline, so the only thing that the creator of the library need to provide is a wrapper for handling communication with a test harness (through pipeline).

Is there a better approach?

Answer 1

I think that's the right approach. The bulk of the test will be written exactly once, and then a piece of the test will be implemented for each implementation of the library. And then the test will talk to the language-specific piece via some communication mechanism, e.g. a socket or a file.