Sean M. Beatty says in "Where testing fails" article that "deadlocks, stack overflows, race conditions and timing problems cannot be detected by testing (whether it is code inspection, whide-box structural and black-box functional)". Beatty, therefore, recalls some special but well-known methods to detect these specific issues.

How do you call the proposed activity if not testing?
Why is it excluded from the test?

3 Answers 3


This is a terminology issue: what Beatty is saying is that conventional testing methods are unable to detect those conditions.

Essentially, they don't manifest in typical testing activities (and detection requires detailed analysis of the code base by someone with access to and knowledge of the code - which many testers lack). Certainly in my career I've never been in a position to detect any of those conditions short of stumbling across them by chance.

"Testing" in Beatty's article is used to mean "conventional testing" - Security testing falls under the same umbrella as the activities Beatty proposes to track down deadlocks and the like: a specialist activity which someone employed as a tester probably isn't qualified to perform.

Hopefully that helps explain things.


actually the article starts by saying "Testing, alone, cannot find all potential failure points", I don't think anyone here can argue with that- we don't presume to catch all failures in any system.

The proposed activity is called "Code Review"

  • Isn't code review considered as a part of static testing? Is it different from formal verification? Do you define testing as something that does not detect the bugs? But let me +1 for pointing out that this was "code review"
    – Val
    Apr 17, 2013 at 12:33
  • Code review might be defined as static testing, but usually developers are responsible for doing it (unless you work at Microsoft where testers participate in code reviews)
    – Rsf
    Apr 18, 2013 at 6:46
  • Actually, this is a little more detailed than the typical code review - not that this is a bad thing! It's more along the lines of detailed code analysis.
    – Kate Paulk
    Apr 18, 2013 at 12:35
  • @KatePaulk, what do you refer by "it"?
    – Val
    Apr 22, 2013 at 9:19
  • Val, by "it" I mean the techniques in Beatty's article. Sorry for being unclear.
    – Kate Paulk
    Apr 22, 2013 at 11:43

I totally disagree. Running the same test on the same machine will not do much but running integration tests and running on different environments you will surface some of the racing conditions (works especially for desktop apps)

Another great way to iron out race conditions and the such is to perform load testing - works especially for services and websites. The extra load will force the threat scheduler to adopt different patters in allocating threads. For database level deadlocks you can monitor performance counters that relate to this. For the service level - well you'll most likely get some exceptions that are very difficult to reproduce - I recommend taking a look at a historical debugger such as IntelliTrace if you are in the .net world.

Also, check out a research product from Microsoft called Chess - it essentially changes the thread scheduler of a process on every test run - that should trigger quite a few of the race conditions.


  • The integration tests detect worst cases at random, right? Whereas the code analysis offered is systematic. "Systematic techniques, deemed to be more comprehensive and capable to find important corner cases that would be likely to be overlooked by random techniques." (loose.upt.ro/~oose/uploads/RTSS/rtss-lecture5.pdf)
    – Val
    Apr 19, 2013 at 6:26
  • I've read the document you refer to. It is very general and focused on code-level testing, which is not appropriate for a specialized QA nor is it effective in commercial products (not talking about mission-critical systems). Devs should do unit testing and code reviews. Automatic code analysis depends on the capabilities of the language - Java / C# are much better than C++ and good luck with javascript and co. I have not seen manual code analysis performed other than code reviews. Apr 19, 2013 at 9:12
  • I see now that CHESS is attributed to the model checkers category of verification, which particularly addresses the deadlock detection. Thanks for that reference. So Beatty now seems to exclude the whole domain of formal verification from the "testing" domain.
    – Val
    Apr 19, 2013 at 9:54
  • Not really sure what you mean in your last sentence, could you reformulate? Apr 21, 2013 at 12:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.