35

As I understand it, this sentence means that automated tests don't find new bugs in existing code (though they might catch a new bug in new code that breaks old code). Is this correct? Or is this piece of wisdom actually claiming that regression tests rarely catch regressions in practice?

Also, is this statement wrong for automated exploratory testing (e.g., fuzzing, randomized inputs to data-driven tests) that can end up running specific tests that were never run before and finding unexpected exceptions, hangs, or program crashes that indicate bugs? Would it be more accurate to say that automated checking will not find new bugs in old code? Or am I still missing the point of this statement?

This seems like a commonly cited piece of tester wisdom, so I really feel like I need to understand both what the statement is getting at, and any exceptions to its correctness.

  • Can you tell us where you read, or heard, this? – Mawg Jun 30 '17 at 11:50
  • No, it was years ago at this point. Couldn't track it down w/ a quick Google. But it was floating around in multiple places at the time, IIRC. It seemed to be coming from the same places that wanted to refer to automated "checks" instead of "automated tests", etc., so I think it might have been part of the pro-manual / pro-exploratory pushback to the high levels of automated testing starting to take over the industry (and pushback to the corresponding devaluing of manual or exploratory testing skill). – Ethel Evans Jul 3 '17 at 21:49

10 Answers 10

30

If test automation does the job it is intended to, when a defect is introduced, it should be found and fixed very quickly, hopefully within a single daily cycle.

If this is how the automation suite is working, then automation should become a "barrier" that prevents new bugs from being created in code that is already working and tested.

When you think of it like that, a good working automation suite will find few, if any, new bugs, but it should prevent new bugs from being introduced.

Indirectly, however, automation should free up testers so they can perform other exploratory and manual testing activities, which will find bugs.

So, test automation does in fact allow new bugs to be found, just not in the way you would expect.

On fuzzing, I haven't seen that type of testing in widespread use, and it doesn't feel very efficient to me (other than in a security testing context). I consider the use of test data generators, using equivalence methods and boundary testing to be more effective in finding issues.

  • Thanks, I wasn't getting the idea that the bug 'isn't a bug' if it's fixed so quickly that it doesn't really spend any time in the code. Do you have more information / good links to read on 'test data generators'? I'm trying to figure out if the people I've worked with have used the term 'fuzzing' loosely when they were doing what you are talking about at least some of the time, or if there is a greater distinction that I'm just not familiar with. When I've seen people 'fuzz' in the past, they did have x% of the tests be category A, y% category B, etc., where A & B are defined by a regex. – Ethel Evans May 10 '11 at 23:15
  • "Fuzzing" commonly refers to automatically generating invalid, unexpected, or random data. Truly random data, may not find any bugs, ever, however semi-random data can be quite effective. Sorry, I can't think of any links off hand, maybe I need to write a blog post. – Bruce McLeod May 10 '11 at 23:40
  • Thanks, Bruce, I thought that was the distinction you were aiming at. I appreciate your patience in clarifying these terms for me. I think the concepts got conflated because testers so naturally added in semi-random data with their random testing (since it's a fairly small set of changes to the fuzzing tool to have it run semi-random data as well). – Ethel Evans May 11 '11 at 16:49
  • For an example of how a modern fuzz testing tool works, see: en.wikipedia.org/wiki/American_fuzzy_lop_(fuzzer) – gmatht Jul 2 '17 at 6:48
20

I don't like the statement because it assumes that the only thing you can automate are regression tests. If you rewrote the statment as "Regression tests don't find new bugs", then I think it's accurate.

I often write automation (e.g. performance, data driven, or model-based tests) that find new bugs - and I think the automation effort is incomplete without some sort of automation beyond regression testing.

  • 4
    In all honesty, every place I've worked has dropped the word regression and called what you would call "automated regression tests" simply "automated tests." Whether it be proper or not, I find it to be fairly common. – corsiKa May 14 '11 at 3:29
  • 5
    IMO, that's sad. – Alan May 14 '11 at 3:45
  • 1
    I wouldn't say it's any sadder than any of the other misconceptions and mis-defined words and phrases in software development. For a set of people who make or break programs based on the tiniest change, it's surprising not only what we are willing to accept as incorrect, but often refuse to change even when corrected. – corsiKa May 14 '11 at 4:51
  • It's been years, but I occasionally review old questions I asked and their answers - usually when a new answer is added - and this particular answer still makes me smile because I agree with it so strongly. Thought you might like to know that this is still affecting someone even six years later :-) – Ethel Evans Nov 29 '17 at 22:10
  • Thanks for saying this - it means a lot to me that I can make sense sometimes. – Alan Dec 18 '17 at 21:07
8

I think there's one more piece to the answer that I see being hinted at in others' responses, but not directly called out.

Often, automated tests are created in such a way that even non-essential details are locked in and executed the same way. This leads to what has been referred to as the Pesticide Paradox or the minefield analogy. Bugs are found when the test is created, and then the test effectiveness at finding new bugs drops as the test does the exact same thing over and over every time without variance. In some cases, you might want this behavior - barricade tests like Bruce describes, regression tests designed to look for the recurrence of certain data-specific bugs, and such. For many people, it seems like this is all automation is which is where the conventional "wisdom" of automated tests not finding new bugs comes from. In this scenario, the test's greatest bug finding effectiveness is at creation time. After that, as bugs gets fixed, it moves more to a "providing ongoing confidence" model. It may occasionally find a new bug that gets introduced, but barring any direct changes in that part of the application, it may never find further problems.

It is possible, however, to create tests that extend their bug finding effectiveness past their initial creation. By thinking about each test and identifying what is critical to the test and what is only specified for convenience or repeatability, you can take advantage of the non-critical aspects for the test to provide variance.

For example, I work on a healthcare information system. A test for registering a new patient obviously needs to actually register a patient. We might have separate tests for registering a patient through the hospital admittance, through the emergency room, or through pre-scheduling an admittance. However, depending on the test, it might not matter whether the patient is male or female, or born before 1930, or all sorts of other aspects that we'd probably say shouldn't matter. We could create automation that uses a given patient record to add, and have it do the same thing each time, perhaps even having done some analysis of test data beforehand to manually vary some aspects of the patient across the suite, but this single test would always do the same thing with the same data. We could also, however, hook the test up to a fake data generator and get broader variance in this particular test scenario to catch things beyond what we initially identified. This method would likely take more coding - you'd need to get the expected data values to the places where the verification occurs in addition to the places where it's entered, handle the variance throughout the code, and if repeatability is important in the context of this test, you'd need some way to either reuse a specific random seed or create a log file with an executable version of the test with hard-coded data or something like that, but the test would have the potential to find more bugs. In this case, each time you run the test it has the potential to find a new bug because it may be a "new" test in some way. We might be right and the things we're varying really don't matter, and in that case that we still won't find any new bugs. If we're wrong though, and something does matter, we have a better chance of finding it than if we never vary anything.

  • I can't help thinking about poor Henry Null in your healthcare information system. :) There really is no totally effective way to separate test-only data from real data, because any time there is it just means you haven't tested with fully real data. – Wildcard Jul 1 '17 at 4:21
6

The theory behind it is that an automated test will only catch exactly what it tests for. Since someone had to code it manually, someone is aware of how it will fail. By this definition it's not a new bug - someone already knows about it when they write the test.

One could argue that it can catch new bugs, but not new symptoms. A lot of it depends on semantics.

I would also contend that your point about fuzzy testing is contrary to standard testing philosophy: if you run the same test with the same input, it should have the same output. A fuzzy/random test will not satisfy this condition. That's not to say it may not give useful information, but it does not fall under the umbrella of what people typically refer to when they say "automated testing."

  • The statement, "If you run a test with the same input, it should have the same output" breaks down for other types of testing besides fuzzing - stability testing often is "run until it breaks", testing features with inherent randomness can't have this feature (e.g., I'm testing a random sampling feature right now), etc. Someone distinguished between "testing" and "checking" once, but I can't find the articles - but I thought that was a useful distinction. – Ethel Evans May 11 '11 at 16:53
  • 1
    @Ethel As @Alan pointed out, there is an implicit "Regression" in there. It is my understanding that when most people talk about "automated testing" they are referring to "automated regression testing." I may be mistaken on this, and everyone's experience will be different, but in instances I can recall where it has come up, automated testing meant regression testing unless it was specifically qualified otherwise. – corsiKa May 11 '11 at 17:04
  • yeah, I'm picking that up. I'm finding it frustrating, because automated testing can and should be thought of as being - at least potentially - so much more. Anyways, I think I've found a new pet peeve :) Time to go laugh at myself! – Ethel Evans May 11 '11 at 17:37
4

In my experience, I tend to find more bugs while I'm creating the automated tests than I find by executing them.

Still, it's not true that automated tests don't find bugs - some automated tests find some bugs, even if it's not many. And done well, the cost of executing automated tests is low enough that it's worth doing.

I don't view the automation choice as a binary one. I often automate some testing in a project, while leaving some of the testing manual. The trick is in determining what is worth automating, and what is not. When the choice is made reasonably well, but manual and automated tests find bugs. When the choice is made extremely well, each method findd bugs that would be more difficult (or more expensive) to find using the other method.

1

Part of the wisdom here comes from the fact that the automated test shouldn't find anything new, otherwise you didn't test the code before creating them. The only "new" bugs an automated test should find are the ones that 1 tester can't do manually by themselves, such as some multi-user stress tests.

  • But . . . I usually didn't test the code before creating them. It's pretty rare that the code was finished before I made my tests. I thought that parallel development of tests and code was standard? – Ethel Evans May 11 '11 at 17:55
  • Not always. You might be working on a legacy app, where most of the code is completely untested, or you might be working on a waterfall project where the "testing phase" doesn't start until the "development phase" is done. – testerab May 28 '11 at 23:07
  • I don't think that multi-user bugs are the only bugs to be found. Any back-end change/error, multi browser issues etc can also be found by automated test! – Nawshad Rehan Rasha Nov 2 '17 at 7:10
1

Automated test find one thing, feature, or unit of the product or codebase. It doesn't actually try and automatically find new features of the codebase or new code paths. It's the test creators responsibility to design tests for new features and code paths.

1

I think the answer is that

automated tests don't find new bugs

should really be stated as

pre-existing automated tests don't find bugs in newly developed features

As you can see it doesn't roll off the tongue quite so easily. In fact, even worse it should probably be

pre-existing automated tests don't find new bugs in newly developed features and also may not find new bugs in modified features

ugh !

1

Common wisdom or common sense might be faulty, like this one. Is rule valuable if it works 80% of the time? Yes it is (in 80% of the situations :-) ), if you are aware that it is not 100% true. It allows you to focus your attention.

Automated tests find bugs all the time (it just did it for me last week). If they did not, nobody would be using them.

But as Michael Durrant answer explains, it is bit more complicated.

Automated "test" are wrong name: they are automated "checks", where script is performing known actions and checking for known resulting conditions. Such checks can check only condition which programmer anticipated - anything not anticipated will be ignored.

So i.e. if code is checking for presence of 2 known error messages, it will ignore the third, the new one. If test is checking for presence of text span with CSS class error-message, new error will be detected (if it uses correct CSS class).

Good automated test can check (some) new kinds on new errors, detect new bugs.

0

Automated tests serve effectively for multi-browser testing where old code may introduce new bugs and those can be caught. Because latest versions can cause compatibility issue with old stable code.

Also for multi-user and load testing, new bugs can be found with the help of automated tests.

Even if system environment or configuration related any changes happens and new bugs introduce, those can be marked by this automated checks.

So, "automated tests don't find new bugs"- is not always right! But honest confession is- most of the time it is.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.