Is a distinction between “testing” and “checking” useful?

Question

I've seen testers (Michael Bolton, mainly) make a distinction between "testing" (looking for defects using a variety of different methods) and "checking" (running a scenario and then verifying a set of facts which, if false, imply a defect in the product). Under these definitions, checking is a subset of testing.

Examples: Most automated regression tests would be "checking". Manual testing that doesn't go "off the rails" would be manual checking, but as soon as a tester leaves a script and starts investigating other behavior, they've started exploratory testing. Ad-hoc testing would not be checking. Fuzzing and stability testing would normally not be "checks", but automated exploratory tests.

I feel like there are a number of cases where people say, "Tests should have feature X", but then there are 10 exceptions to the rule. However, many of these statements seem to be truer and clearer when stated as "Checks should have feature X". For example, "Given the same inputs, automated checks should always produce the same outputs unless there is a change to the code." However, there are many tests (stability, performance, semi-random input, etc.) that would naturally have some level of variation in their outputs that are useful but do not meet this criteria. However, I also can see how this distinction could seem overly pedantic.

Concluding question, or, TL;DR: Is the idea of "checking" as a subset of testing a useful one?

P.S. Edits to link articles / sources for this concept will be appreciated, if anyone here knows what I'm talking about.

P.P.S. What I'm trying to get at is, 'Is a distinction between "testing" vs. "checking" like this useful', not 'Is it used?' I know it's not a commonly used concept, but it is a concept that people have been trying to introduce. Does the concept of "checking" to describe a subset of "testing" help with communication about tests and testing? Would it be a good thing to introduce it to tester / QA vocabulary?

Answer 1

I would like to emphasize that usefulness (like quality) is a subjective and personal/organizational thing. In other words, the general question "Is it useful?" needs context, of which some person(s) is the most important part. I made the distinction because I found that people lump checking into testing in a way that I found *un*helpful.

For example, in DuncN's situation above, people see checking as part of testing. That's cool, and I agree. The trouble starts when people see checking as equivalent to testing, which it is manifestly not, unless you decide to simplify testing to the degree that it loses most of its information value. As my colleague James Bach likes to point out, it's as though people treated compiling--which can be automated--as the only interesting and useful part of programming. Anyone who knows something about programming will tell you that compiling is probably the least interesting part about programming. I argue that, similarly, running checks is the least interesting part of testing. Yet some people seem fascinated by it.

Here's the story on that: http://www.developsense.com/blog/2009/08/testing-vs-checking/ and http://www.developsense.com/blog/2009/11/merely-checking-or-merely-testing/

@Bruce...

The reason so many people who think seriously about testing reject the ISTQB is because its material fails to make distinctions like this.

Instead of letting the ISTQB tell you whether something is useful or not, you could check it out for yourself and make your own evaluation. Then, if it's not useful to you, that's totally cool.

---Michael B.

Answer 2

The distinction between testing and checking adds no significant value to the profession of testing beyond witty philosophical banter around the water-cooler, or lead to changes in our SDLC processes.

Perhaps the concept of checks vs. test may add value if it causes testers to reflect on the purpose or intent of a test. But, would that lead to a change in our professional jargon? The distinction is trivial at best IMHO.

I agree that "Testing is something we do with the motivation of finding new information." Sometimes even "exploratory testing" doesn't reveal new information that is important to the stakeholders, and sometimes well-desiggned automated test will reveal new information that was previously undiscovered via other testing approaches. Also, some tests (that might otehrwise be called checks) will reveal new information when changes in the product occur.

Testing is not about "trying things" and then pondering whether or not the outcome is a problem. Boris Beizer stated, "In kiddie testing the tester says, after the fact, the the observed outcome of the test was (or was not) the expected outcome. In real testing the outcome is predicted..." In other words, when I am testing I should at least be able to predict to some degree what will occur or what state I will be in after I perform some set of actions; otherwise we are just guessing "is there a problem here?" So, if someone really wants to make a distinction between checking and testing, then they should also make a distinction between testing and guessing. I sometimes see an awful lot of guessing going on.

Answer 3

Most often I hear these terms when distinguishing between someone who is a 'Sapient Tester' meaning they understand a lot of test theory, and can create their own test cases etc (often on the fly as with exploratory testing) and someone who is merely following a script someone else has written and 'checking' the results (and would perhaps miss any bug that occurred during this process if it was not spelled out as one of the results to check for. To me the former person is someone capable of doing real 'testing' while the latter is mostly just 'human based test automation'

I've sometimes heard (and used the term myself) the latter referred to as 'monkey testers' in that their skill level is little better than a trained chimp. However I think that's not really a good term as it could be construed to be racist (I don't think it is, or at least I've never intended it to be such when I've used it. It is a bit insulting perhaps, but really should not be seen as an insult to the individual, (we all have to start somewhere) but rather to the org that hires such people for 'testing'. I've run across a few folks who claim the title of a 'test professional' but they've been given no education or training by their employer, nor sought it on their own, and have basically just been doing 'checking' for as much as 2 years and honestly think they understand QA.

I think the racial implications come about because a lot of people employed in the 'checker' role seem to be working for offshore test outsourcing companies, and thus an unfair stereotype starts to emerge as a result.

I have run across people who have been working as checkers but are interested in learning test theory and to actually start using their brains and their wits to do 'real' testing. It's rather gratifying actually and I'm glad to see an increasing movement of like minded folks getting together and self educating (since their employers seem to have no interest in it) themselves to learn real testing, and start being able to create their own test cases, understand what tests are appropriate when/where etc.

I've helped a few here and there with some mentoring, and I know folks like Michael Bolton and James Bach have been very generous with their time helping such groups get started and teaching folks to apply critical thinking to the test process, instead of just running checklists created by others.

(sorry for the long diatribe, but I think it's actually a pretty important topic)

Answer 4

I believe that I understand what you're talking about. To myself at least, a check(s) is most definitely a part of a test.

The check determines if the system is producing the desired output given a certain input under in a specific state. These checks are used to aid in your decision if the test passed or failed.

Stability tests, performance tests, and the like rarely say that the test passed or failed as it would typically return a set of values for a human to read and make the decision of whether the test passed or failed as the automation that you are using can rarely determine the state of the environment in which it is in. Say that you are running a performance test for a web application. With a very small code change, you see that page load times have increased by 500ms. According to your automated checks, that may seem as though it is a catastrophic failure. When you look at the results however and ask about the state of the server that it was on, you find out that another team was running a load test on another intensive application on the same server. In itself, this may also not provide any details, however, it would aid in your final judgement after running the performance test again and comparing the results.

Not really the same anaology at all, but, I usually like to reference the following from Michael Bolton which is very similar to what you are describing.

Answer 5

I agree with Lyndon on the Michael Bolton site. He's the biggest of advocate of "Testing" vs "Checking" I've come across.

Michael Boltons blog is immense with loads of references - he has some great posts & you will spend hours in there, especially if you follow all the references.

Its an idea I'm peddling at my current client, and most people seem to be understand the difference, but why bother having it. In the large, the term "Testing" is felt to encompass the term "Checking" - checking is a part of testing

Answer 6

I am not familiar with the use of the term "checks" in common QA parlence, and it is not listed in the ISTQB glossary, so I would say that the answer to your question is no.

That said, I would say that how you use the term would be in reference to the act of performing vaidation of expeced and actual results, whilst executing a test case.

Answer 7

Although we don't use the word "checking", in my current shop we do make a distinction between the two concepts.

We have one batch system where we concentrate only on the output. We never look at the code used to create the output, it could be (and often is) changed at any time between batch runs. And when we find an issue, it's considered perfectly acceptable to manually modify the output, rather than fix the code. To me this fits in with what is being referred to as checking.

For all of our other systems, we do more of what is being called testing. We drive the code and modules through a series of tests, and look everywhere - not just at the output. We would never accept manual fixing of the output as a solution to a bug.

To us, this distinction is useful.

Answer 8

I find it valuable to distinguish between testing and checking. Maybe not as passionately or pedantically as some. :) I think it is most important when you start talking about automation. Automation can check a lot of things. So people talk about replacing hands-on (and brain-on) testing with automation. But there's an extra level of human thought and experimentation that I have yet to see captured effectively in an automation suite. For this reason, I find it valuable to remind automation addicts that their checks are useful, but we also need to test.