Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Sean M. Beatty says in "Where testing fails" article that "deadlocks, stack overflows, race conditions and timing problems cannot be detected by testing (whether it is code inspection, whide-box structural and black-box functional)". Beatty, therefore, recalls some special but well-known methods to detect these specific issues.

How do you call the proposed activity if not testing?
Why is it excluded from the test?

share|improve this question
add comment

3 Answers 3

up vote 7 down vote accepted

This is a terminology issue: what Beatty is saying is that conventional testing methods are unable to detect those conditions.

Essentially, they don't manifest in typical testing activities (and detection requires detailed analysis of the code base by someone with access to and knowledge of the code - which many testers lack). Certainly in my career I've never been in a position to detect any of those conditions short of stumbling across them by chance.

"Testing" in Beatty's article is used to mean "conventional testing" - Security testing falls under the same umbrella as the activities Beatty proposes to track down deadlocks and the like: a specialist activity which someone employed as a tester probably isn't qualified to perform.

Hopefully that helps explain things.

share|improve this answer
add comment

actually the article starts by saying "Testing, alone, cannot find all potential failure points", I don't think anyone here can argue with that- we don't presume to catch all failures in any system.

The proposed activity is called "Code Review"

share|improve this answer
    
Isn't code review considered as a part of static testing? Is it different from formal verification? Do you define testing as something that does not detect the bugs? But let me +1 for pointing out that this was "code review" –  Val Apr 17 '13 at 12:33
    
Code review might be defined as static testing, but usually developers are responsible for doing it (unless you work at Microsoft where testers participate in code reviews) –  Rsf Apr 18 '13 at 6:46
    
Actually, this is a little more detailed than the typical code review - not that this is a bad thing! It's more along the lines of detailed code analysis. –  Kate Paulk Apr 18 '13 at 12:35
    
@KatePaulk, what do you refer by "it"? –  Val Apr 22 '13 at 9:19
    
Val, by "it" I mean the techniques in Beatty's article. Sorry for being unclear. –  Kate Paulk Apr 22 '13 at 11:43
add comment

I totally disagree. Running the same test on the same machine will not do much but running integration tests and running on different environments you will surface some of the racing conditions (works especially for desktop apps)

Another great way to iron out race conditions and the such is to perform load testing - works especially for services and websites. The extra load will force the threat scheduler to adopt different patters in allocating threads. For database level deadlocks you can monitor performance counters that relate to this. For the service level - well you'll most likely get some exceptions that are very difficult to reproduce - I recommend taking a look at a historical debugger such as IntelliTrace if you are in the .net world.

Also, check out a research product from Microsoft called Chess - it essentially changes the thread scheduler of a process on every test run - that should trigger quite a few of the race conditions.

http://research.microsoft.com/en-us/projects/chess/

share|improve this answer
    
The integration tests detect worst cases at random, right? Whereas the code analysis offered is systematic. "Systematic techniques, deemed to be more comprehensive and capable to find important corner cases that would be likely to be overlooked by random techniques." (loose.upt.ro/~oose/uploads/RTSS/rtss-lecture5.pdf) –  Val Apr 19 '13 at 6:26
    
I've read the document you refer to. It is very general and focused on code-level testing, which is not appropriate for a specialized QA nor is it effective in commercial products (not talking about mission-critical systems). Devs should do unit testing and code reviews. Automatic code analysis depends on the capabilities of the language - Java / C# are much better than C++ and good luck with javascript and co. I have not seen manual code analysis performed other than code reviews. –  Bogdan Gavril Apr 19 '13 at 9:12
    
I see now that CHESS is attributed to the model checkers category of verification, which particularly addresses the deadlock detection. Thanks for that reference. So Beatty now seems to exclude the whole domain of formal verification from the "testing" domain. –  Val Apr 19 '13 at 9:54
    
Not really sure what you mean in your last sentence, could you reformulate? –  Bogdan Gavril Apr 21 '13 at 12:18
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.