For example, let's pretend that I want to write a program that encrypts data using a given algorithm.

Instead of manually calculating expected outputs and comparing them with actual outputs, I might instead compare the outputs produced by my program with outputs produced by OpenSSL.

There was a term for this but I can't remember. What is the name of this practice?


3 Answers 3


As mentioned in my comment, you are probably referring to a special type of test oracle.

João Farias' answer already got to the point: test oracles are used all the time. However, the type of the oracle varies. When you, e.g., write a unit test for a particular method:

// 1st param: expected, 2nd param: actual
assertEquals(3, mySumMethod(1, 2));

The expected output is typically determined by a human—the developer. But sometimes, you refer to a reference implementation:

assertEquals(referenceSumMethod(1, 2), mySumMethod(1, 2));

In contrast to the first variant, the human oracle, I would say this falls into the category of the derived oracle. As far as I'm aware, this is sometimes used in property-based testing with libs like QuickCheck, for example.

For further reading: In Douglas Hoffman's paper "A Taxonomy for Test Oracles", you find various classes of test oracles. But also Elaine J. Weyuker's original work on the oracle assumption ("On Testing Non-testable Programs") is a nice read.

  • "In contract to the first variant".. you mean contrast I presume?
    – Ray Oei
    Nov 12, 2018 at 22:22
  • @RayOei oh, yeah – typo! Thanks for hint, fixed it.
    – beatngu13
    Nov 12, 2018 at 22:24
  • Thanks. In "On Testing Non-testable Programs" they are also called pseudo-oracles. Nov 13, 2018 at 0:35

I've often heard of this as regression testing.

  • You know how things worked before and tests pass
  • You make a change
  • You test again and see if anything has regressed with bugs
  • While regression testing indeed uses a derived oracle—a previous version of the SUT, which is why Hoffman calls this consistency oracle since it compares the consistency between two versions—I think that it is an important difference whether you use your own implementation or a different system to verify your test results.
    – beatngu13
    Nov 12, 2018 at 22:32
  • I agree it is usually to compare against a previous state that you control. I think it could also be applied (with note) against a known state in another system. Thanks for teaching me about the oracle term. Nov 13, 2018 at 0:17

It would say it depends on your goal with the test:

Portability testing and compatibility testing mean that your system will work on different environments.

However, your goal maybe to do security testing because you trust the encryption from OpenSSL.

You can also say you are doing automated test data generation or something like this, because you are externalizing completely the test data generation to computers.

About @beatngu13 answer: A test oracle is used all the time. Oracles are models to check the outputs. If you were to calculate the expected outputs, the oracle would be your mental model of the system. Using OpenSSL, your oracle is the OpenSSL model. That's why I think it doesn't answer properly the question.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.