Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

As I understand it, this sentence means that automated tests don't find new bugs in existing code (though they might catch a new bug in new code that breaks old code). Is this correct? Or is this piece of wisdom actually claiming that regression tests rarely catch regressions in practice?

Also, is this statement wrong for automated exploratory testing (e.g., fuzzing, randomized inputs to data-driven tests) that can end up running specific tests that were never run before and finding unexpected exceptions, hangs, or program crashes that indicate bugs? Would it be more accurate to say that automated checking will not find new bugs in old code? Or am I still missing the point of this statement?

This seems like a commonly cited piece of tester wisdom, so I really feel like I need to understand both what the statement is getting at, and any exceptions to its correctness.

share|improve this question
add comment

7 Answers 7

up vote 21 down vote accepted

If test automation does the job it is intended to, when a defect is introduced, it should be found and fixed very quickly, hopefully within a single daily cycle.

If this is how the automation suite is working, then automation should become a "barrier" that prevents new bugs from being created in code that is already working and tested.

When you think of it like that, a good working automation suite will not find few, if any new bugs, but it should prevent new bugs from being introduced.

Indirectly, however, automation should free up testers so they can perform other, exploratory and manual testing activities which will find bugs.

So, test automation does infact allow new bugs to be found, just not in the way you would expect.

On fuzzing, I haven't seen that type of testing in widespread use, and it doesn't feel very effecient to me (other than in a security testing context). I think that the use of test data generators, using equivalence methods and boundary testing to be more effective in finding issues.

share|improve this answer
    
Thanks, I wasn't getting the idea that the bug 'isn't a bug' if it's fixed so quickly that it doesn't really spend any time in the code. Do you have more information / good links to read on 'test data generators'? I'm trying to figure out if the people I've worked with have used the term 'fuzzing' loosely when they were doing what you are talking about at least some of the time, or if there is a greater distinction that I'm just not familiar with. When I've seen people 'fuzz' in the past, they did have x% of the tests be category A, y% category B, etc., where A & B are defined by a regex. –  Ethel Evans May 10 '11 at 23:15
    
"Fuzzing" commonly refers to automatically generating invalid, unexpected, or random data. Truly random data, may not find any bugs, ever, however semi-random data can be quite effective. Sorry, I can't think of any links off hand, maybe I need to write a blog post. –  Bruce McLeod May 10 '11 at 23:40
    
Thanks, Bruce, I thought that was the distinction you were aiming at. I appreciate your patience in clarifying these terms for me. I think the concepts got conflated because testers so naturally added in semi-random data with their random testing (since it's a fairly small set of changes to the fuzzing tool to have it run semi-random data as well). –  Ethel Evans May 11 '11 at 16:49
add comment

I think there's one more piece to the answer that I see being hinted at in others' responses, but not directly called out.

Often, automated tests are created in such a way that even non-essential details are locked in and executed the same way. This leads to what has been referred to as the Pesticide Paradox or the minefield analogy. Bugs are found when the test is created, and then the test effectiveness at finding new bugs drops as the test does the exact same thing over and over every time without variance. In some cases, you might want this behavior - barricade tests like Bruce describes, regression tests designed to look for the recurrence of certain data-specific bugs, and such. For many people, it seems like this is all automation is which is where the conventional "wisdom" of automated tests not finding new bugs comes from. In this scenario, the test's greatest bug finding effectiveness is at creation time. After that, as bugs gets fixed, it moves more to a "providing ongoing confidence" model. It may occasionally find a new bug that gets introduced, but barring any direct changes in that part of the application, it may never find further problems.

It is possible, however, to create tests that extend their bug finding effectiveness past their initial creation. By thinking about each test and identifying what is critical to the test and what is only specified for convenience or repeatability, you can take advantage of the non-critical aspects for the test to provide variance.

For example, I work on a healthcare information system. A test for registering a new patient obviously needs to actually register a patient. We might have separate tests for registering a patient through the hospital admittance, through the emergency room, or through pre-scheduling an admittance. However, depending on the test, it might not matter whether the patient is male or female, or born before 1930, or all sorts of other aspects that we'd probably say shouldn't matter. We could create automation that uses a given patient record to add, and have it do the same thing each time, perhaps even having done some analysis of test data beforehand to manually vary some aspects of the patient across the suite, but this single test would always do the same thing with the same data. We could also, however, hook the test up to a fake data generator and get broader variance in this particular test scenario to catch things beyond what we initially identified. This method would likely take more coding - you'd need to get the expected data values to the places where the verification occurs in addition to the places where it's entered, handle the variance throughout the code, and if repeatability is important in the context of this test, you'd need some way to either reuse a specific random seed or create a log file with an executable version of the test with hard-coded data or something like that, but the test would have the potential to find more bugs. In this case, each time you run the test it has the potential to find a new bug because it may be a "new" test in some way. We might be right and the things we're varying really don't matter, and in that case that we still won't find any new bugs. If we're wrong though, and something does matter, we have a better chance of finding it than if we never vary anything.

share|improve this answer
add comment

In my experience, I tend to find more bugs while I'm creating the automated tests than I find by executing them.

Still, it's not true that automated tests don't find bugs - some automated tests find some bugs, even if it's not many. And done well, the cost of executing automated tests is low enough that it's worth doing.

I don't view the automation choice as a binary one. I often automate some testing in a project, while leaving some of the testing manual. The trick is in determining what is worth automating, and what is not. When the choice is made reasonably well, but manual and automated tests find bugs. When the choice is made extremely well, each method findd bugs that would be more difficult (or more expensive) to find using the other method.

share|improve this answer
add comment

Automated test find one thing, feature, or unit of the product or codebase. It doesn't actually try and automatically find new features of the codebase or new code paths. It's the test creators responsibility to design tests for new features and code paths.

share|improve this answer
add comment

Part of the wisdom here comes from the fact that the automated test shouldn't find anything new, otherwise you didn't test the code before creating them. The only "new" bugs an automated test should find are the ones that 1 tester can't do manually by themselves, such as some multi-user stress tests.

share|improve this answer
    
But . . . I usually didn't test the code before creating them. It's pretty rare that the code was finished before I made my tests. I thought that parallel development of tests and code was standard? –  Ethel Evans May 11 '11 at 17:55
    
Not always. You might be working on a legacy app, where most of the code is completely untested, or you might be working on a waterfall project where the "testing phase" doesn't start until the "development phase" is done. –  testerab May 28 '11 at 23:07
add comment

I don't like the statement because it assumes that the only thing you can automate are regression tests. If you rewrote the statment as "Regression tests don't find new bugs", then I think it's accurate.

I often write automation (e.g. performance, data driven, or model-based tests) that find new bugs - and I think the automation effort is incomplete without some sort of automation beyond regression testing.

share|improve this answer
2  
In all honesty, every place I've worked has dropped the word regression and called what you would call "automated regression tests" simply "automated tests." Whether it be proper or not, I find it to be fairly common. –  corsiKa May 14 '11 at 3:29
2  
IMO, that's sad. –  Alan May 14 '11 at 3:45
    
I wouldn't say it's any sadder than any of the other misconceptions and mis-defined words and phrases in software development. For a set of people who make or break programs based on the tiniest change, it's surprising not only what we are willing to accept as incorrect, but often refuse to change even when corrected. –  corsiKa May 14 '11 at 4:51
add comment

The theory behind it is that an automated test will only catch exactly what it tests for. Since someone had to code it manually, someone is aware of how it will fail. By this definition it's not a new bug - someone already knows about it when they write the test.

One could argue that it can catch new bugs, but not new symptoms. A lot of it depends on semantics.

I would also contend that your point about fuzzy testing is contrary to standard testing philosophy: if you run the same test with the same input, it should have the same output. A fuzzy/random test will not satisfy this condition. That's not to say it may not give useful information, but it does not fall under the umbrella of what people typically refer to when they say "automated testing."

share|improve this answer
    
The statement, "If you run a test with the same input, it should have the same output" breaks down for other types of testing besides fuzzing - stability testing often is "run until it breaks", testing features with inherent randomness can't have this feature (e.g., I'm testing a random sampling feature right now), etc. Someone distinguished between "testing" and "checking" once, but I can't find the articles - but I thought that was a useful distinction. –  Ethel Evans May 11 '11 at 16:53
    
@Ethel As @Alan pointed out, there is an implicit "Regression" in there. It is my understanding that when most people talk about "automated testing" they are referring to "automated regression testing." I may be mistaken on this, and everyone's experience will be different, but in instances I can recall where it has come up, automated testing meant regression testing unless it was specifically qualified otherwise. –  corsiKa May 11 '11 at 17:04
    
yeah, I'm picking that up. I'm finding it frustrating, because automated testing can and should be thought of as being - at least potentially - so much more. Anyways, I think I've found a new pet peeve :) Time to go laugh at myself! –  Ethel Evans May 11 '11 at 17:37
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.