Why bother writing/executing a test whose results will never be used?

Question

A discussion with friends recently wandered on to the topic of tests that are executed yet whose results are never used. Why bother running them? Why bother even writing them?

Thinking of them we all agreed is useful - we do that all the time, think of a test we could run, and then decide not to run it for any number of reasons.

Actually executing it, though - or even write it out - seems less useful.

One of us said "Well, auditors might want to see the results", to which I replied "Then you're using the results, aren't you?"

Someone else said "You might need the results later, so you run the test now just in case", to which I replied "YAGNI" (You Aren't Going To Need It).

The Rule of Three states that there are at least three distinct reasons to run such a test; I'm trying to puzzle out what they might be. Any ideas?

Answer 1

-You are under contract to run N tests, or to run this particular test (but not provide results).

-The testing equivalent of a filibuster. You know that if you do too much testing you will find more bugs and jeopardize the release so you run tests with no value.

-So that someone can learn the system in a zero consequence environment. I don't think this is the best way to learn a system, but I have seen this done.

Answer 2

This is a very strange question. Surely, if we accept your premise that the test results will never be used -- never used by anyone, for any reason, at any point in time -- we must conclude that there is no point in running the test.

I do not know why you posted this question. What kind of reply do expect from us? "You may want to run the test because it is winter and the heat has gone out and the only way to stay warm is by running this test while holding your hands over the CPU." "No", you will say, "the heat generated by the CPU is a kind of test result, and I already said the test results will never be used, therefore you cannot warm your hands this way."

Then someone else will answer, "Perhaps you are at war with Iran, and you know the computers that run their nuclear fuel reprocessing plant run on battery power, and this test uses lots of CPU time, so if you run the test on their computers, you can exhaust the batteries, shut down their nuclear reprocessing plant, prevent them from building nuclear weapons, and win the war.". And you will reply, "Exhausting the batteries is a kind of test result, and I told you the test results will never be used, therefore you cannot exhaust the batteries this way."

Perhaps you expect someone to answer, "What if I want to run the test just so that it produces results?" and you will say, "No, if you have the desire to produce the results then actually producing them is a way to satisfy that desire, and therefore is a way to use the results, and I just said you cannot do that."

Of course, it is conceivable that your premise is wrong. Perhaps it would be a better to give an example of a test whose results you believe will never be used, and ask us whether we agree with you.

Answer 3

I'd think of this as a progression:

1. You write a test because you have a reason, if this reason does not exist you would not write it
2. You run a test because you have a result to obtain, if this result is not needed you would not run it
3. You report on a test because the data is required, if this data is not reported on then you would not include it

If you have any of these three, which is what I use to determine a test, then you can always go back to the step that caused you to write or run or report on the test. If those demands have changed, and that happens, then the test might be obsolete but at one time it was needed for something.

Answer 4

Executing tests in which the results are never reported occur all the time with exploratory testing, bug bashes, dog-fooding, etc.

Numerous tests executed while exploring are used at least by the person doing the testing, unless of course the person is simply "guessing."

Answer 5

If the tests are relevant enough they might do you good after you run them and they fail.

I would image that after they fail the results WILL be interesting.

I had a bunch of auto tests who were doing a bunch of tests of which nobody cared about. It always sent green results. Everyday or so.

Then one day a red email arrived. Image the trouble it caused ;-)

So run them until they fail. If they don't fail then at least be happy that you in your heart covered these possibilities too.

Answer 6

What does "results are never used" actually mean?

If you executed a test and it crashed the system, you would write a bug report, wouldn't you? In that case, the result is used. If the test executed and found no errors, then you have used the result to decide not to write a bug report. In both cases, the results are indeed used.

You execute a test only because you are interested in the results. If you somehow are certain that you are not interested in the results - no matter what those results might be - then don't execute the test.