We have a logging protocol that defines how output of logging libraries should look like, e.g.:

  • each line should be a JSON
  • each line should not exceed 1MB

There are multiple implementation of this logging protocol, i.e, logging libraries written in different languages (Java, Python, etc.). I would like to create one set of tests that I can run against any logging library to verify it complies to the logging protocol. I don't want authors of a new library to port existing test. Instead, I want them to reuse existing tests.

The are already known solutions to that like Reactive Streams Test Compatibility Kit, but they work for implementations in the same language (or at least for JVM-languages). In my case implementations can be of different languages.

How can I make tests language independent?

Initial idea

  1. My first idea is based on assumption that logging libraries always take one or two arguments:

    logging.error("Serious problem occurred", exception)

    and log output to a log file

    { level: "error", message: "Serious problem occurred", stacktrace: "....."}
  2. The test harness would pass some string to the testing library and verify the output.

  3. The test harness would communicate with a logging library through a pipeline, so the only thing that the creator of the library need to provide is a wrapper for handling communication with a test harness (through pipeline).

Is there a better approach?

  • Do those different implementation have unit tests? Can you run tests for one implementation against another? Who is the "owner" of the standard, and do they define some "authoritative" tests which any implementation should meet? Nov 13 '17 at 17:42
  • @PeterMasiar Some of those implementations have. However, they are in different languages, so there's no direct way to run punit tests (Python) against Java implementation. The owner of the standard is another team. They do not define tests but rather rules that implementations must abide. Those rules I plan to translate to tests.
    – dzieciou
    Nov 14 '17 at 12:32

I think that's the right approach. The bulk of the test will be written exactly once, and then a piece of the test will be implemented for each implementation of the library. And then the test will talk to the language-specific piece via some communication mechanism, e.g. a socket or a file.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.