Heuristics for identifying "one-time" tests?

Question

I understand the importance of having good "regression tests" to ensure fixed bugs don't get reintroduced. But I think it's possible to go too far and end up with a massive "regression test suite" for every bug you've fixed, every feature that's been changed, even every feature that's been removed...you get the idea. Are there any heuristics or techniques for identifying when a test belongs in the long-term regression suite, versus when it would be better to write a "one-time" or "throwaway" test that is run for the next release, and then discarded? Ideally, we would like to identify this at the time we are authoring them, though tips for identifying what to prune out over time are helpful as well.

The main concern is minimizing the time required to test each release. We have a large legacy codebase that uses old technology that isn't very conducive to automated testing, which means much of our regression testing is manual. We don't want to keep growing our manual testing burden for each release with tests that aren't adding long-term value.

We're also in a regulated environment, so we do need some way of documenting what we tested, and I think there's value in designing a good test when you change something rather than just doing ad hoc testing, but some things, realistically, aren't going to suddenly break two years down the road if we demonstrate, now, that they are fixed.

Some examples that my "gut" says are only useful for the first release that includes them:

• Changing the shape of a graphical control display and ensuring all the places that show such a display use the new shape (i.e. we didn't repeat ourselves and duplicate the code for the display--this is an example from our legacy codebase, where we often found copy-pasted code and other repetition).
• For a software product that controls hardware, we removed support for some add-on devices that the hardware supported, where such devices are stored in a database. We tested that upgrading the database from the version that supported the devices to the version that no longer supported them behaved appropriately, and that the devices were no longer present in the new version.
• Fixing an "obvious" bug where a dialog didn't clean up properly when canceling, etc.

To ask the same question in the other direction:

How do you keep a manual regression suite from "ballooning" out of control over time?

Answer 1

I'd look at a number of things when deciding whether to add something to a regression suite or not. A lot of what I'd consider would be very product specific, though; understanding how clients use the product, what the upgrade behavior is, and so on would be very important here.

1) What is the risk if the bug the test is designed to detect is missed?
Depending at the level the test happens (unit test, integration test, validation test, etc) this may be hard to estimate, but the closer you are to what the end user will see, the easier it will probably be to tell what the risk is.

2) Where in the program hierarchy is the code change that's being tested? With very bad spaghetti code, this may be hard to tell, but the more things that a function is relied on by, the fewer things you (probably) need to test related to it, as it breaking should break any number of other components of the OS.

3) How many things have/had to go wrong for the bug to expose itself? Here, I'd actually tend to keep things that involve more things going wrong in the regression suite than things that expose themselves more easily, because you're more likely to find the simple bugs in unrelated testing than you are to find the more complicated bugs. (This is assuming you have testers who will notice bugs that happen that aren't part of the test scenario documentation.)

3) Does/can the test encompass other tests? As an example, when I was first testing an area of new, complicated code, I first verified that function A worked. Then I verified that function B worked. Then I verified that function C worked. Those functions all were in the same area of code, and were related functionality, so I wanted to be able to narrow down the source of the error if a problem happened, and make debugging easier. Now I test them all at the same time, because the tests are all important, but I'm surprised if they find a problem. And if one of them does, and the debugging ends up being difficult, I can always go back and test the functions individually to see which one is the problem.

4) How long has the bug in question been in the code, and what exposed it? This one is hard to quantify without knowing more about what's being tested, but if you had something that was exposed specifically because of an edge case, I wouldn't necessarily add it to a regression bucket, but I would add it to my list of "silly things end-users do that would have never occurred to me" that I should consider when writing tests in the future.

5) How subtle would a problem in some area be? The more subtle the problem, the more testing you should do around it.

6) How often is some function going to happen in the field? In the example you gave above, the database updating, that may only need to be tested the once, if you can be sure that everyone is going to get the new code as soon as it's released, before you release a new version. However, if you might have someone next pick up a new version of the program in three years, who currently has a version that's two years out of date, then not including the upgrade scenario in your regression bucket might be a problem. (I'm not sure how you do migration testing now, so this may or may not be something you worry about).

Answer 2

One approach that can be taken is to write the one throw away test that tests the bug in detail, but add a single step or two to an existing test in the regression suite that validates that area still looks okay. It would still increase the suite, but at a much smaller rate.

'Automated tests' are ambiguous, it has a lot of meaning (Selenium end to end tests, integration tests, unit tests, etc..).

Assuming you mean some sort of 'End to end, front end tests', another approach is to have development write a unit test around this area that validates the bug is fixed (which is run in a build pipeline), manually test it once, and then archive the manual test.

Answer 3

Seems that question was substantially edited since I wrote my answer. OP now says they have huge suite of manual tests.

One rule of thumb I read somewhere was to consider discarding any test which did not detected a bug in a year. Discard it if the issue is not critical, keep it if detected issue is critical.

---

If you want to run test only once, I do not see a point to automate it. Just do it manually.

If you run test multiple times, why you fell you need to discard it? CPU time is cheap.

Only discard it when requirements changed and it is not worth fixing to accommodate changed requirements.

Code (including automated tests) is like a tattoo: it will stick on you for a LONG time.

If you want to perform some test just few times, strongly consider NOT automating it 100%: prepare test document which sapient human can follow. Only consider automating parts which will be repeated many (dozens and hundreds) of times. Or where exact steps and timing is crucial.

Answer 4

Some considerations:

• set a time limit, when the suite reaches it, reduce suite size until it is under the limit
• set up the manual test to mock and stub whenever possible and suitable
• tag tests and perform subsets such as happy and smoke
• big picture view. 1 test may fail rarely, as in 1 in 1000. With 1000 tests, prob = 1:1 ok?

Answer 5

I don't understand this. Are you testing manually modules that were not changed because they are included in regression tests? Then don't do that. Regression test areas where there is a risk that something might go wrong.

Or you change something in one module, and there is a possibility that something breaks in another module so you have to test that too? Then you need better development practices, because that is not alright, and tests wont help much, and it is meaningless to even discuss testing practices.