Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I've seen testers (Michael Bolton, mainly) make a distinction between "testing" (looking for defects using a variety of different methods) and "checking" (running a scenario and then verifying a set of facts which, if false, imply a defect in the product). Under these definitions, checking is a subset of testing.

Examples: Most automated regression tests would be "checking". Manual testing that doesn't go "off the rails" would be manual checking, but as soon as a tester leaves a script and starts investigating other behavior, they've started exploratory testing. Ad-hoc testing would not be checking. Fuzzing and stability testing would normally not be "checks", but automated exploratory tests.

I feel like there are a number of cases where people say, "Tests should have feature X", but then there are 10 exceptions to the rule. However, many of these statements seem to be truer and clearer when stated as "Checks should have feature X". For example, "Given the same inputs, automated checks should always produce the same outputs unless there is a change to the code." However, there are many tests (stability, performance, semi-random input, etc.) that would naturally have some level of variation in their outputs that are useful but do not meet this criteria. However, I also can see how this distinction could seem overly pedantic.

Concluding question, or, TL;DR: Is the idea of "checking" as a subset of testing a useful one?

P.S. Edits to link articles / sources for this concept will be appreciated, if anyone here knows what I'm talking about.

P.P.S. What I'm trying to get at is, 'Is a distinction between "testing" vs. "checking" like this useful', not 'Is it used?' I know it's not a commonly used concept, but it is a concept that people have been trying to introduce. Does the concept of "checking" to describe a subset of "testing" help with communication about tests and testing? Would it be a good thing to introduce it to tester / QA vocabulary?

share|improve this question
    
I've never hear the "checking" term used in that way, but the concept is reasonable. I prefer the terms "positive testing" (trying to use the app) and "negative testing" (trying to break the app). –  user246 May 18 '11 at 23:20
add comment

8 Answers 8

up vote 10 down vote accepted

I would like to emphasize that usefulness (like quality) is a subjective and personal/organizational thing. In other words, the general question "Is it useful?" needs context, of which some person(s) is the most important part. I made the distinction because I found that people lump checking into testing in a way that I found *un*helpful.

For example, in DuncN's situation above, people see checking as part of testing. That's cool, and I agree. The trouble starts when people see checking as equivalent to testing, which it is manifestly not, unless you decide to simplify testing to the degree that it loses most of its information value. As my colleague James Bach likes to point out, it's as though people treated compiling--which can be automated--as the only interesting and useful part of programming. Anyone who knows something about programming will tell you that compiling is probably the least interesting part about programming. I argue that, similarly, running checks is the least interesting part of testing. Yet some people seem fascinated by it.

Here's the story on that: http://www.developsense.com/blog/2009/08/testing-vs-checking/ and http://www.developsense.com/blog/2009/11/merely-checking-or-merely-testing/

@Bruce...

The reason so many people who think seriously about testing reject the ISTQB is because its material fails to make distinctions like this.

Instead of letting the ISTQB tell you whether something is useful or not, you could check it out for yourself and make your own evaluation. Then, if it's not useful to you, that's totally cool.

---Michael B.

share|improve this answer
    
Thanks a ton for stopping by and posting! This is very helpful. –  Ethel Evans May 19 '11 at 1:13
    
@Michael ... Since that answer, I am now VERY familiar with the term, although until I had come across your material, I wasn't at all. –  Bruce McLeod Nov 23 '11 at 3:33
    
This is just me being nit-picky but rather than using compiling as your dev analogy to checking I would use running a wizard to generate boilerplate code. –  Ron Pihlgren Nov 27 '11 at 6:05
add comment

The distinction between testing and checking adds no significant value to the profession of testing beyond witty philosophical banter around the water-cooler, or lead to changes in our SDLC processes.

Perhaps the concept of checks vs. test may add value if it causes testers to reflect on the purpose or intent of a test. But, would that lead to a change in our professional jargon? The distinction is trivial at best IMHO.

I agree that "Testing is something we do with the motivation of finding new information." Sometimes even "exploratory testing" doesn't reveal new information that is important to the stakeholders, and sometimes well-desiggned automated test will reveal new information that was previously undiscovered via other testing approaches. Also, some tests (that might otehrwise be called checks) will reveal new information when changes in the product occur.

Testing is not about "trying things" and then pondering whether or not the outcome is a problem. Boris Beizer stated, "In kiddie testing the tester says, after the fact, the the observed outcome of the test was (or was not) the expected outcome. In real testing the outcome is predicted..." In other words, when I am testing I should at least be able to predict to some degree what will occur or what state I will be in after I perform some set of actions; otherwise we are just guessing "is there a problem here?" So, if someone really wants to make a distinction between checking and testing, then they should also make a distinction between testing and guessing. I sometimes see an awful lot of guessing going on.

share|improve this answer
    
Thanks, this is a great response. I like your point about guessing. –  Ethel Evans May 27 '11 at 17:28
add comment

Most often I hear these terms when distinguishing between someone who is a 'Sapient Tester' meaning they understand a lot of test theory, and can create their own test cases etc (often on the fly as with exploratory testing) and someone who is merely following a script someone else has written and 'checking' the results (and would perhaps miss any bug that occurred during this process if it was not spelled out as one of the results to check for. To me the former person is someone capable of doing real 'testing' while the latter is mostly just 'human based test automation'

I've sometimes heard (and used the term myself) the latter referred to as 'monkey testers' in that their skill level is little better than a trained chimp. However I think that's not really a good term as it could be construed to be racist (I don't think it is, or at least I've never intended it to be such when I've used it. It is a bit insulting perhaps, but really should not be seen as an insult to the individual, (we all have to start somewhere) but rather to the org that hires such people for 'testing'. I've run across a few folks who claim the title of a 'test professional' but they've been given no education or training by their employer, nor sought it on their own, and have basically just been doing 'checking' for as much as 2 years and honestly think they understand QA.

I think the racial implications come about because a lot of people employed in the 'checker' role seem to be working for offshore test outsourcing companies, and thus an unfair stereotype starts to emerge as a result.

I have run across people who have been working as checkers but are interested in learning test theory and to actually start using their brains and their wits to do 'real' testing. It's rather gratifying actually and I'm glad to see an increasing movement of like minded folks getting together and self educating (since their employers seem to have no interest in it) themselves to learn real testing, and start being able to create their own test cases, understand what tests are appropriate when/where etc.

I've helped a few here and there with some mentoring, and I know folks like Michael Bolton and James Bach have been very generous with their time helping such groups get started and teaching folks to apply critical thinking to the test process, instead of just running checklists created by others.

(sorry for the long diatribe, but I think it's actually a pretty important topic)

share|improve this answer
add comment

I believe that I understand what you're talking about. To myself at least, a check(s) is most definitely a part of a test.

The check determines if the system is producing the desired output given a certain input under in a specific state. These checks are used to aid in your decision if the test passed or failed.

Stability tests, performance tests, and the like rarely say that the test passed or failed as it would typically return a set of values for a human to read and make the decision of whether the test passed or failed as the automation that you are using can rarely determine the state of the environment in which it is in. Say that you are running a performance test for a web application. With a very small code change, you see that page load times have increased by 500ms. According to your automated checks, that may seem as though it is a catastrophic failure. When you look at the results however and ask about the state of the server that it was on, you find out that another team was running a load test on another intensive application on the same server. In itself, this may also not provide any details, however, it would aid in your final judgement after running the performance test again and comparing the results.

Not really the same anaology at all, but, I usually like to reference the following from Michael Bolton which is very similar to what you are describing.

share|improve this answer
    
Thanks a ton for the Michael Bolton link, which is what I was getting at. I'll throw it up in my question. –  Ethel Evans May 12 '11 at 18:00
add comment

I agree with Lyndon on the Michael Bolton site. He's the biggest of advocate of "Testing" vs "Checking" I've come across.

Michael Boltons blog is immense with loads of references - he has some great posts & you will spend hours in there, especially if you follow all the references.

Its an idea I'm peddling at my current client, and most people seem to be understand the difference, but why bother having it. In the large, the term "Testing" is felt to encompass the term "Checking" - checking is a part of testing

share|improve this answer
add comment

I am not familiar with the use of the term "checks" in common QA parlence, and it is not listed in the ISTQB glossary, so I would say that the answer to your question is no.

That said, I would say that how you use the term would be in reference to the act of performing vaidation of expeced and actual results, whilst executing a test case.

share|improve this answer
    
Bruce, have you lost the last part of your sentence above, or is it just a typo of a comma for a full stop? I can read the sentence as is, but it seems incomplete. –  testerab May 13 '11 at 1:03
    
yeah ... quick answer and I borked the ending ... I can't remember what else was there :-( –  Bruce McLeod May 13 '11 at 5:19
add comment

Although we don't use the word "checking", in my current shop we do make a distinction between the two concepts.

We have one batch system where we concentrate only on the output. We never look at the code used to create the output, it could be (and often is) changed at any time between batch runs. And when we find an issue, it's considered perfectly acceptable to manually modify the output, rather than fix the code. To me this fits in with what is being referred to as checking.

For all of our other systems, we do more of what is being called testing. We drive the code and modules through a series of tests, and look everywhere - not just at the output. We would never accept manual fixing of the output as a solution to a bug.

To us, this distinction is useful.

share|improve this answer
add comment

I find it valuable to distinguish between testing and checking. Maybe not as passionately or pedantically as some. :) I think it is most important when you start talking about automation. Automation can check a lot of things. So people talk about replacing hands-on (and brain-on) testing with automation. But there's an extra level of human thought and experimentation that I have yet to see captured effectively in an automation suite. For this reason, I find it valuable to remind automation addicts that their checks are useful, but we also need to test.

share|improve this answer
    
Not all automation is rudimentary script-lets, and some automation designs incorporate things such as data variations and or state transition variations (model based testing or "automated exploration") that involves 'experimentation.' –  Bj Rollison Nov 17 '11 at 16:38
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.