What is the difference between "automated testing" and "automated regression testing"?

Question

"Automated Testing" as a term seems to have a number of different uses. For some, it means built in unit tests that, when a new build is run, execute to make sure code refactors have not broken something. That is not at question here.

What seems to be the question, though, is the difference between automated tests that are used to execute the initial tests against a program/product/feature for initial verification and automated tests that are run against a program/product/feature to make sure that nothing that had been working before is now broken. I know I kind of answered my own question for the definition of the two, but what I'm looking for is practical characteristic differences. What would you do in an "automated test" that you wouldn't do in an "automated regression test" and vice versa? ARE there any differences? Does one lead into another?

Answer 1

The truth is for a lot of people, there isn't a difference.

A regression test is something that is going to check your test results against some requirements. Good examples of this are unit tests and functional tests. They tell you whether or not your application has regressed in its functionality (hence, regression.) Of course that assumes it was properly functional in the first place, but the point is still valid. (Note: someone may correct me and say functional tests aren't regression tests. I would be perfectly fine if they did and explained why they do or don't qualify!)

There are other types of automated tests: fuzzy inputs and automated exploratory testing come to mind. These are not always going to have predictable, repeatable outcomes like your regression tests are. They would be a fairly tedious and time consuming effort for a tester to do manually, so there's nothing wrong with automating it and viewing the result. It helps put the limit of testing into how fast you can think, and then automate, as opposed to how fast you can click.

Most people I talk to, especially those in development, consider "automated tests" to mean "automated regression tests" and they honestly don't realize it.

Answer 2

To supplement glowcoder's answer, mainly with more examples, automated regression tests are a subset of automated testing, but there are other kinds. Some examples of non-regression automated tests are:

• Any automated test that will be thrown away and not used to test for regressions is not a regression test, including unit tests that are not used later for regression testing. Technically, any automated test written for new functionality is not a regression test until it has verified correct functionality at least once. Only then can it verify that the functionality hasn't changed.
• Reliability testing, where you have your system under test run in a near-production environment as long as possible to see how long it will run without errors or crashes
• Stress testing, which is about seeing how the system under test behaves under heavy use or limited resources; you might remove CPUs or reduce bandwidth, or simulate a large number of users.
• Some performance testing may not have specific checks to verify and might need a person to interpret the results, although some performance tests can be run as regression tests (e.g., "Run 100 iterations of this function. If it took less than 5 seconds, pass; else, fail.").
• Glowcoder mentioned fuzz testing, which is randomized inputs to a function, but the topic deserves some expansion. With fuzz testing, the tester is looking for unexpected behaviors that could cause problems, such as security holes, crashes, and hangs.

I'm making this a community wiki, so people can add other examples.

Answer 3

Some good answers added already, but I haven't seen this distinction yet, so I'll add it to the pile:

You may automate some tests for initial verification of the build against requirements - but another way of describing that is that you may decide to conduct specification workshops, with the goal of distilling Key Examples, that you can then turn into Executable Specifications.

Gojko Adzic describes this much better in the blog I linked above - but for us, the key difference here is that this automation is meant to illustrate the requirements rather than to test them. So rather than trying to get a lot of coverage of different permutations as you might with a regression suite, you are deliberately keeping the set of examples smaller.

I guess another way of looking is that if you're familiar with the Agile Test Quadrants, then it's the distinction between business facing tests that support the team, and business facing tests that critique the product.

Further info: http://www.acceptancetesting.info/the-book http://specificationbyexample.com/

Answer 4

Typically when regression testing is talked about, it's tests that ensure features don't break from one release to another.

Automated testing is a big category. It includes, but is not limited to regression testing with it's sub categories like smoke/bvt testing, unit tests, scripted UI automation. There are lots of other kinds of automation. Monkey automation that just randomly pushes buttons and looks for crashes, randomized fuzz testing, data integrity testing of databases and semi automated tasks are some examples. There are many more

Further, the traits of good regression tests are different than other categories of automation. Regression test code needs to be relatively fast, reliable, diagnostic and easy to maintain. The goal is not to find "good" or new bugs. The goal is to ensure the changes to the code base don't have a ripple effect that isn't notices early on. Your regression tests should always be passing. If a test is failing every day for a week and no one cares you have a problem. Soon it will be two, then ten and then no one will care about the regression reports because "those things never pass anyway."

There is a large class of ad-hoc automation and semi-automated tests. These should be cheap, planned to have a short lifespan and be super useful in either finding new bugs or pointing out areas where bugs might be lurking. Simply parsing log files with a perl script or logparse.exe and looking for anomalies is a good example. Because the results often require human judgment to parse, writing super clean automation to find bugs that may or may not exist has a poor return on investment. It's possible to find entire classes of bugs, which then need a root cause and some focused regression tests very cheaply. Don't apply the rigid coding standards to throw away testing. Rule of thumb, if it's faster to just write the script again next release than to document and check it in, throw your junky code away and make a note to do it again next release.

Performance and Stress automation are not often considered regression automation. You can make your perf and stress automation into regression tests, but it's not always cost effective. They often end up in the semi-automatic camp where a person has to curate the tests and the results for each run.

Lastly there is a infrastructure automation. Deployment scripts, database cleanup jobs and the regression test framework are examples. They often fall into a gray area between test and product. Some of the best infrastructure automation has built in build-test-deploy cycles. None of that is regression testing per se, but having good infrastructure automation is a key to a smooth process.

Answer 5

A regression testing is somewhat we perform on the application as soon as it undergoes a code change to ensure that new code has not affected other parts of the software. But automated testing is a wide concept of which regression can be the part of. I think a comparison on this is like not possible. However, we can have the same for regression testing and re-testing. You can look at this difference over here in this article.