What is meant by "Automated tests don't find new bugs"?

Question

As I understand it, this sentence means that automated tests don't find new bugs in existing code (though they might catch a new bug in new code that breaks old code). Is this correct? Or is this piece of wisdom actually claiming that regression tests rarely catch regressions in practice?

Also, is this statement wrong for automated exploratory testing (e.g., fuzzing, randomized inputs to data-driven tests) that can end up running specific tests that were never run before and finding unexpected exceptions, hangs, or program crashes that indicate bugs? Would it be more accurate to say that automated checking will not find new bugs in old code? Or am I still missing the point of this statement?

This seems like a commonly cited piece of tester wisdom, so I really feel like I need to understand both what the statement is getting at, and any exceptions to its correctness.

Answer 1

If test automation does the job it is intended to, when a defect is introduced, it should be found and fixed very quickly, hopefully within a single daily cycle.

If this is how the automation suite is working, then automation should become a "barrier" that prevents new bugs from being created in code that is already working and tested.

When you think of it like that, a good working automation suite will find few, if any, new bugs, but it should prevent new bugs from being introduced.

Indirectly, however, automation should free up testers so they can perform other exploratory and manual testing activities, which will find bugs.

So, test automation does in fact allow new bugs to be found, just not in the way you would expect.

On fuzzing, I haven't seen that type of testing in widespread use, and it doesn't feel very efficient to me (other than in a security testing context). I consider the use of test data generators, using equivalence methods and boundary testing to be more effective in finding issues.

Answer 2

I don't like the statement because it assumes that the only thing you can automate are regression tests. If you rewrote the statment as "Regression tests don't find new bugs", then I think it's accurate.

I often write automation (e.g. performance, data driven, or model-based tests) that find new bugs - and I think the automation effort is incomplete without some sort of automation beyond regression testing.

Answer 3

I think there's one more piece to the answer that I see being hinted at in others' responses, but not directly called out.

Often, automated tests are created in such a way that even non-essential details are locked in and executed the same way. This leads to what has been referred to as the Pesticide Paradox or the minefield analogy. Bugs are found when the test is created, and then the test effectiveness at finding new bugs drops as the test does the exact same thing over and over every time without variance. In some cases, you might want this behavior - barricade tests like Bruce describes, regression tests designed to look for the recurrence of certain data-specific bugs, and such. For many people, it seems like this is all automation is which is where the conventional "wisdom" of automated tests not finding new bugs comes from. In this scenario, the test's greatest bug finding effectiveness is at creation time. After that, as bugs gets fixed, it moves more to a "providing ongoing confidence" model. It may occasionally find a new bug that gets introduced, but barring any direct changes in that part of the application, it may never find further problems.

It is possible, however, to create tests that extend their bug finding effectiveness past their initial creation. By thinking about each test and identifying what is critical to the test and what is only specified for convenience or repeatability, you can take advantage of the non-critical aspects for the test to provide variance.

For example, I work on a healthcare information system. A test for registering a new patient obviously needs to actually register a patient. We might have separate tests for registering a patient through the hospital admittance, through the emergency room, or through pre-scheduling an admittance. However, depending on the test, it might not matter whether the patient is male or female, or born before 1930, or all sorts of other aspects that we'd probably say shouldn't matter. We could create automation that uses a given patient record to add, and have it do the same thing each time, perhaps even having done some analysis of test data beforehand to manually vary some aspects of the patient across the suite, but this single test would always do the same thing with the same data. We could also, however, hook the test up to a fake data generator and get broader variance in this particular test scenario to catch things beyond what we initially identified. This method would likely take more coding - you'd need to get the expected data values to the places where the verification occurs in addition to the places where it's entered, handle the variance throughout the code, and if repeatability is important in the context of this test, you'd need some way to either reuse a specific random seed or create a log file with an executable version of the test with hard-coded data or something like that, but the test would have the potential to find more bugs. In this case, each time you run the test it has the potential to find a new bug because it may be a "new" test in some way. We might be right and the things we're varying really don't matter, and in that case that we still won't find any new bugs. If we're wrong though, and something does matter, we have a better chance of finding it than if we never vary anything.

Answer 4

The theory behind it is that an automated test will only catch exactly what it tests for. Since someone had to code it manually, someone is aware of how it will fail. By this definition it's not a new bug - someone already knows about it when they write the test.

One could argue that it can catch new bugs, but not new symptoms. A lot of it depends on semantics.

I would also contend that your point about fuzzy testing is contrary to standard testing philosophy: if you run the same test with the same input, it should have the same output. A fuzzy/random test will not satisfy this condition. That's not to say it may not give useful information, but it does not fall under the umbrella of what people typically refer to when they say "automated testing."

Answer 5

In my experience, I tend to find more bugs while I'm creating the automated tests than I find by executing them.

Still, it's not true that automated tests don't find bugs - some automated tests find some bugs, even if it's not many. And done well, the cost of executing automated tests is low enough that it's worth doing.

I don't view the automation choice as a binary one. I often automate some testing in a project, while leaving some of the testing manual. The trick is in determining what is worth automating, and what is not. When the choice is made reasonably well, but manual and automated tests find bugs. When the choice is made extremely well, each method findd bugs that would be more difficult (or more expensive) to find using the other method.

Answer 6

Part of the wisdom here comes from the fact that the automated test shouldn't find anything new, otherwise you didn't test the code before creating them. The only "new" bugs an automated test should find are the ones that 1 tester can't do manually by themselves, such as some multi-user stress tests.

Answer 7

I think the answer is that

automated tests don't find new bugs

should really be stated as

pre-existing automated tests don't find bugs in newly developed features

As you can see it doesn't roll off the tongue quite so easily. In fact, even worse it should probably be

pre-existing automated tests don't find new bugs in newly developed features and also may not find new bugs in modified features

ugh !

Answer 8

Automated test find one thing, feature, or unit of the product or codebase. It doesn't actually try and automatically find new features of the codebase or new code paths. It's the test creators responsibility to design tests for new features and code paths.

Answer 9

Common wisdom or common sense might be faulty, like this one. Is rule valuable if it works 80% of the time? Yes it is (in 80% of the situations :-) ), if you are aware that it is not 100% true. It allows you to focus your attention.

Automated tests find bugs all the time (it just did it for me last week). If they did not, nobody would be using them.

But as Michael Durrant answer explains, it is bit more complicated.

Automated "test" are wrong name: they are automated "checks", where script is performing known actions and checking for known resulting conditions. Such checks can check only condition which programmer anticipated - anything not anticipated will be ignored.

So i.e. if code is checking for presence of 2 known error messages, it will ignore the third, the new one. If test is checking for presence of text span with CSS class error-message, new error will be detected (if it uses correct CSS class).

Good automated test can check (some) new kinds on new errors, detect new bugs.

Answer 10

Automated tests serve effectively for multi-browser testing where old code may introduce new bugs and those can be caught. Because latest versions can cause compatibility issue with old stable code.

Also for multi-user and load testing, new bugs can be found with the help of automated tests.

Even if system environment or configuration related any changes happens and new bugs introduce, those can be marked by this automated checks.

So, "automated tests don't find new bugs"- is not always right! But honest confession is- most of the time it is.