I understand the importance of having good "regression tests" to ensure fixed bugs don't get reintroduced. But I think it's possible to go too far and end up with a massive "regression test suite" for every bug you've fixed, every feature that's been changed, even every feature that's been removed...you get the idea. Are there any heuristics or techniques for identifying when a test belongs in the long-term regression suite, versus when it would be better to write a "one-time" or "throwaway" test that is run for the next release, and then discarded? Ideally, we would like to identify this at the time we are authoring them, though tips for identifying what to prune out over time are helpful as well.
The main concern is minimizing the time required to test each release. We have a large legacy codebase that uses old technology that isn't very conducive to automated testing, which means much of our regression testing is manual. We don't want to keep growing our manual testing burden for each release with tests that aren't adding long-term value.
We're also in a regulated environment, so we do need some way of documenting what we tested, and I think there's value in designing a good test when you change something rather than just doing ad hoc testing, but some things, realistically, aren't going to suddenly break two years down the road if we demonstrate, now, that they are fixed.
Some examples that my "gut" says are only useful for the first release that includes them:
- Changing the shape of a graphical control display and ensuring all the places that show such a display use the new shape (i.e. we didn't repeat ourselves and duplicate the code for the display--this is an example from our legacy codebase, where we often found copy-pasted code and other repetition).
- For a software product that controls hardware, we removed support for some add-on devices that the hardware supported, where such devices are stored in a database. We tested that upgrading the database from the version that supported the devices to the version that no longer supported them behaved appropriately, and that the devices were no longer present in the new version.
- Fixing an "obvious" bug where a dialog didn't clean up properly when canceling, etc.
To ask the same question in the other direction:
How do you keep a manual regression suite from "ballooning" out of control over time?