5

For example, let's pretend that I want to write a program that encrypts data using a given algorithm.

Instead of manually calculating expected outputs and comparing them with actual outputs, I might instead compare the outputs produced by my program with outputs produced by OpenSSL.

There was a term for this but I can't remember. What is the name of this practice?

3

As mentioned in my comment, you are probably referring to a special type of test oracle.

João Farias' answer already got to the point: test oracles are used all the time. However, the type of the oracle varies. When you, e.g., write a unit test for a particular method:

// 1st param: expected, 2nd param: actual
assertEquals(3, mySumMethod(1, 2));

The expected output is typically determined by a human—the developer. But sometimes, you refer to a reference implementation:

assertEquals(referenceSumMethod(1, 2), mySumMethod(1, 2));

In contrast to the first variant, the human oracle, I would say this falls into the category of the derived oracle. As far as I'm aware, this is sometimes used in property-based testing with libs like QuickCheck, for example.

For further reading: In Douglas Hoffman's paper "A Taxonomy for Test Oracles", you find various classes of test oracles. But also Elaine J. Weyuker's original work on the oracle assumption ("On Testing Non-testable Programs") is a nice read.

| improve this answer | |
  • "In contract to the first variant".. you mean contrast I presume? – Ray Oei Nov 12 '18 at 22:22
  • @RayOei oh, yeah – typo! Thanks for hint, fixed it. – beatngu13 Nov 12 '18 at 22:24
  • Thanks. In "On Testing Non-testable Programs" they are also called pseudo-oracles. – Behrang Saeedzadeh Nov 13 '18 at 0:35
2

I've often heard of this as regression testing.

  • You know how things worked before and tests pass
  • You make a change
  • You test again and see if anything has regressed with bugs
| improve this answer | |
  • While regression testing indeed uses a derived oracle—a previous version of the SUT, which is why Hoffman calls this consistency oracle since it compares the consistency between two versions—I think that it is an important difference whether you use your own implementation or a different system to verify your test results. – beatngu13 Nov 12 '18 at 22:32
  • I agree it is usually to compare against a previous state that you control. I think it could also be applied (with note) against a known state in another system. Thanks for teaching me about the oracle term. – Michael Durrant Nov 13 '18 at 0:17
1

It would say it depends on your goal with the test:

Portability testing and compatibility testing mean that your system will work on different environments.

However, your goal maybe to do security testing because you trust the encryption from OpenSSL.

You can also say you are doing automated test data generation or something like this, because you are externalizing completely the test data generation to computers.

About @beatngu13 answer: A test oracle is used all the time. Oracles are models to check the outputs. If you were to calculate the expected outputs, the oracle would be your mental model of the system. Using OpenSSL, your oracle is the OpenSSL model. That's why I think it doesn't answer properly the question.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.