Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

A discussion with friends recently wandered on to the topic of tests that are executed yet whose results are never used. Why bother running them? Why bother even writing them?

Thinking of them we all agreed is useful - we do that all the time, think of a test we could run, and then decide not to run it for any number of reasons.

Actually executing it, though - or even write it out - seems less useful.

One of us said "Well, auditors might want to see the results", to which I replied "Then you're using the results, aren't you?"

Someone else said "You might need the results later, so you run the test now just in case", to which I replied "YAGNI" (You Aren't Going To Need It).

The Rule of Three states that there are at least three distinct reasons to run such a test; I'm trying to puzzle out what they might be. Any ideas?

share|improve this question

6 Answers 6

Executing tests in which the results are never reported occur all the time with exploratory testing, bug bashes, dog-fooding, etc.

Numerous tests executed while exploring are used at least by the person doing the testing, unless of course the person is simply "guessing."

share|improve this answer
    
I argue the tests are being used, then - because they drive (at least in part) the next thing a person does. –  Michael Hunter Aug 23 '11 at 2:16
    
The distinction between executing a test and reporting the results of a test are important - thanks for reminding me :) Oodles of tests are never reported, I completely agree! –  Michael Hunter Aug 23 '11 at 2:17

I'd think of this as a progression:

  1. You write a test because you have a reason, if this reason does not exist you would not write it
  2. You run a test because you have a result to obtain, if this result is not needed you would not run it
  3. You report on a test because the data is required, if this data is not reported on then you would not include it

If you have any of these three, which is what I use to determine a test, then you can always go back to the step that caused you to write or run or report on the test. If those demands have changed, and that happens, then the test might be obsolete but at one time it was needed for something.

share|improve this answer

-You are under contract to run N tests, or to run this particular test (but not provide results).

-The testing equivalent of a filibuster. You know that if you do too much testing you will find more bugs and jeopardize the release so you run tests with no value.

-So that someone can learn the system in a zero consequence environment. I don't think this is the best way to learn a system, but I have seen this done.

share|improve this answer
    
"Because I'm being paid to go through the motions" and "To keep from doing something else" are two more general classes of things - thanks! –  Michael Hunter Aug 23 '11 at 2:16

This is a very strange question. Surely, if we accept your premise that the test results will never be used -- never used by anyone, for any reason, at any point in time -- we must conclude that there is no point in running the test.

I do not know why you posted this question. What kind of reply do expect from us? "You may want to run the test because it is winter and the heat has gone out and the only way to stay warm is by running this test while holding your hands over the CPU." "No", you will say, "the heat generated by the CPU is a kind of test result, and I already said the test results will never be used, therefore you cannot warm your hands this way."

Then someone else will answer, "Perhaps you are at war with Iran, and you know the computers that run their nuclear fuel reprocessing plant run on battery power, and this test uses lots of CPU time, so if you run the test on their computers, you can exhaust the batteries, shut down their nuclear reprocessing plant, prevent them from building nuclear weapons, and win the war.". And you will reply, "Exhausting the batteries is a kind of test result, and I told you the test results will never be used, therefore you cannot exhaust the batteries this way."

Perhaps you expect someone to answer, "What if I want to run the test just so that it produces results?" and you will say, "No, if you have the desire to produce the results then actually producing them is a way to satisfy that desire, and therefore is a way to use the results, and I just said you cannot do that."

Of course, it is conceivable that your premise is wrong. Perhaps it would be a better to give an example of a test whose results you believe will never be used, and ask us whether we agree with you.

share|improve this answer
    
Thanks for pointing out some missing context in my question; added, hope it helps explain why I'm asking this question. –  Michael Hunter Aug 14 '11 at 0:37
    
I don't think I would say the thing you suggest I'd say. I don't think I'd claim generating heat or exhausting batteries is a kind of test result. –  Michael Hunter Aug 14 '11 at 0:39
1  
Your suggestions do, however, point to an answer: "Because running the test generates desirable side effects, even though the results of the test on the system under test are ignored". –  Michael Hunter Aug 14 '11 at 0:40

What does "results are never used" actually mean?

If you executed a test and it crashed the system, you would write a bug report, wouldn't you? In that case, the result is used. If the test executed and found no errors, then you have used the result to decide not to write a bug report. In both cases, the results are indeed used.

You execute a test only because you are interested in the results. If you somehow are certain that you are not interested in the results - no matter what those results might be - then don't execute the test.

share|improve this answer

If the tests are relevant enough they might do you good after you run them and they fail.

I would image that after they fail the results WILL be interesting.

I had a bunch of auto tests who were doing a bunch of tests of which nobody cared about. It always sent green results. Everyday or so.

Then one day a red email arrived. Image the trouble it caused ;-)

So run them until they fail. If they don't fail then at least be happy that you in your heart covered these possibilities too.

share|improve this answer
    
Sounds like people do actually care about those tests then...and maybe are just pretending nonchalance :) –  Michael Hunter Aug 14 '11 at 0:24

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.