11

I understand the importance of having good "regression tests" to ensure fixed bugs don't get reintroduced. But I think it's possible to go too far and end up with a massive "regression test suite" for every bug you've fixed, every feature that's been changed, even every feature that's been removed...you get the idea. Are there any heuristics or techniques for identifying when a test belongs in the long-term regression suite, versus when it would be better to write a "one-time" or "throwaway" test that is run for the next release, and then discarded? Ideally, we would like to identify this at the time we are authoring them, though tips for identifying what to prune out over time are helpful as well.

The main concern is minimizing the time required to test each release. We have a large legacy codebase that uses old technology that isn't very conducive to automated testing, which means much of our regression testing is manual. We don't want to keep growing our manual testing burden for each release with tests that aren't adding long-term value.

We're also in a regulated environment, so we do need some way of documenting what we tested, and I think there's value in designing a good test when you change something rather than just doing ad hoc testing, but some things, realistically, aren't going to suddenly break two years down the road if we demonstrate, now, that they are fixed.

Some examples that my "gut" says are only useful for the first release that includes them:

  • Changing the shape of a graphical control display and ensuring all the places that show such a display use the new shape (i.e. we didn't repeat ourselves and duplicate the code for the display--this is an example from our legacy codebase, where we often found copy-pasted code and other repetition).
  • For a software product that controls hardware, we removed support for some add-on devices that the hardware supported, where such devices are stored in a database. We tested that upgrading the database from the version that supported the devices to the version that no longer supported them behaved appropriately, and that the devices were no longer present in the new version.
  • Fixing an "obvious" bug where a dialog didn't clean up properly when canceling, etc.

To ask the same question in the other direction:

How do you keep a manual regression suite from "ballooning" out of control over time?

  • I don't actually understand, how are doing these manual regression tests? You basically have a list of things these developers have to check? – Groo Jun 29 '17 at 15:24
  • We literally write a manual test case, or add to an existing one. If it was a bugfix, the test has the tester exercise the functionality that would have exposed the bug, and the expected outcomes describe the correct behavior. Similar for changed features. – c32hedge Jun 29 '17 at 15:27
  • Thanks for the responses so far. One thing that I don't think the current answers have addressed, though, is ideas for identifying when a test is authored whether it will be useful long-term, or just to run once. We definitely need to keep up with pruning out the old ones, but if we had some guidelines in place and could tag "one-time" tests as such when they're authored, that maintenance would be a lot easier. We could also likely be more lightweight in how we write/document them to begin with. – c32hedge Jul 12 '17 at 20:25
  • 2
    It might sound a bit futuristic (however it do not think this is a really complex thing in the modern world) but one could build up a neural netowrk, train it on a current regression set (the most challending thing here is to decompose a test object into a set of "features") and this NN will answer if the test is one-time test on the test creation time. After all NNs are intended to solve classification tasks. – Alexey R. Aug 29 '17 at 11:29
  • @AlexeyR. pretty cool idea but yes decomposing tests into what's being trained on is definitely the challenge. At some point I've wanted to go through and make some kind of "feature map" and correlate that to which areas we're actually testing, so perhaps those ideas could be combined somehow. – c32hedge Aug 30 '17 at 14:09
6
+50

I'd look at a number of things when deciding whether to add something to a regression suite or not. A lot of what I'd consider would be very product specific, though; understanding how clients use the product, what the upgrade behavior is, and so on would be very important here.

1) What is the risk if the bug the test is designed to detect is missed?
Depending at the level the test happens (unit test, integration test, validation test, etc) this may be hard to estimate, but the closer you are to what the end user will see, the easier it will probably be to tell what the risk is.

2) Where in the program hierarchy is the code change that's being tested? With very bad spaghetti code, this may be hard to tell, but the more things that a function is relied on by, the fewer things you (probably) need to test related to it, as it breaking should break any number of other components of the OS.

3) How many things have/had to go wrong for the bug to expose itself? Here, I'd actually tend to keep things that involve more things going wrong in the regression suite than things that expose themselves more easily, because you're more likely to find the simple bugs in unrelated testing than you are to find the more complicated bugs. (This is assuming you have testers who will notice bugs that happen that aren't part of the test scenario documentation.)

3) Does/can the test encompass other tests? As an example, when I was first testing an area of new, complicated code, I first verified that function A worked. Then I verified that function B worked. Then I verified that function C worked. Those functions all were in the same area of code, and were related functionality, so I wanted to be able to narrow down the source of the error if a problem happened, and make debugging easier. Now I test them all at the same time, because the tests are all important, but I'm surprised if they find a problem. And if one of them does, and the debugging ends up being difficult, I can always go back and test the functions individually to see which one is the problem.

4) How long has the bug in question been in the code, and what exposed it? This one is hard to quantify without knowing more about what's being tested, but if you had something that was exposed specifically because of an edge case, I wouldn't necessarily add it to a regression bucket, but I would add it to my list of "silly things end-users do that would have never occurred to me" that I should consider when writing tests in the future.

5) How subtle would a problem in some area be? The more subtle the problem, the more testing you should do around it.

6) How often is some function going to happen in the field? In the example you gave above, the database updating, that may only need to be tested the once, if you can be sure that everyone is going to get the new code as soon as it's released, before you release a new version. However, if you might have someone next pick up a new version of the program in three years, who currently has a version that's two years out of date, then not including the upgrade scenario in your regression bucket might be a problem. (I'm not sure how you do migration testing now, so this may or may not be something you worry about).

  • This is the first answer that has answered from the "test authoring time" perspective in extensive detail. I'm leaving the door open for more good answers in the bounty period but marking this as accepted for now to make sure it gets the bounty if no other answers emerge in the next couple days. – c32hedge Aug 30 '17 at 18:38
  • It's a hard question; if someone else can come up with something better, they're welcome to the bounty. – Kevin McKenzie Aug 30 '17 at 18:54
4

One approach that can be taken is to write the one throw away test that tests the bug in detail, but add a single step or two to an existing test in the regression suite that validates that area still looks okay. It would still increase the suite, but at a much smaller rate.

'Automated tests' are ambiguous, it has a lot of meaning (Selenium end to end tests, integration tests, unit tests, etc..).

Assuming you mean some sort of 'End to end, front end tests', another approach is to have development write a unit test around this area that validates the bug is fixed (which is run in a build pipeline), manually test it once, and then archive the manual test.

  • Assume we're talking about manual tests here and that even unit tests aren't always possible due to poor existing architecture. In the cases where we can write unit tests, we generally do so rather than adding more manual tests. – c32hedge Aug 25 '17 at 19:01
  • +1 for the first point though. It would definitely help if people looked for an existing test to update rather than creating a whole new test case for each regression, but it's been hard to make that happen. – c32hedge Aug 28 '17 at 22:06
3

Seems that question was substantially edited since I wrote my answer. OP now says they have huge suite of manual tests.

One rule of thumb I read somewhere was to consider discarding any test which did not detected a bug in a year. Discard it if the issue is not critical, keep it if detected issue is critical.


If you want to run test only once, I do not see a point to automate it. Just do it manually.

If you run test multiple times, why you fell you need to discard it? CPU time is cheap.

Only discard it when requirements changed and it is not worth fixing to accommodate changed requirements.

Code (including automated tests) is like a tattoo: it will stick on you for a LONG time.

If you want to perform some test just few times, strongly consider NOT automating it 100%: prepare test document which sapient human can follow. Only consider automating parts which will be repeated many (dozens and hundreds) of times. Or where exact steps and timing is crucial.

  • 1
    The question is not about whether to automate or not. As I mentioned, with the technology we're using much of it can't be automated easily. Assume all the tests in question are manual. I 100% agree that CPU time is cheap and if we were automating these tests there'd be no reason to discard, but we're talking tester/developer time so any time we add to testing a release is time we're not spending working on the next release. – c32hedge Jun 28 '17 at 20:35
  • OK you changed your question substantially, so I tried answering your new question. – Peter M. Jun 28 '17 at 21:35
  • All I did was format a few things in bold :) – c32hedge Jun 29 '17 at 13:15
  • The other side of this coin, if a test isn't finding anything, is it the test, or are you not varying how you test, what test data looks like enough to catch things? – Veretax Apr 2 at 19:58
2

Some considerations:

  • set a time limit, when the suite reaches it, reduce suite size until it is under the limit
  • set up the manual test to mock and stub whenever possible and suitable
  • tag tests and perform subsets such as happy and smoke
  • big picture view. 1 test may fail rarely, as in 1 in 1000. With 1000 tests, prob = 1:1 ok?
  • 2
    Could you clarify what you mean by the last bullet point? – c32hedge Jul 11 '17 at 13:53
  • Care to expound? :) – c32hedge Aug 29 '17 at 15:12
1

I don't understand this. Are you testing manually modules that were not changed because they are included in regression tests? Then don't do that. Regression test areas where there is a risk that something might go wrong.

Or you change something in one module, and there is a possibility that something breaks in another module so you have to test that too? Then you need better development practices, because that is not alright, and tests wont help much, and it is meaningless to even discuss testing practices.

  • 1
    I don't quite understand your first question. The manual tests would be the regression tests, but I'd like to get a better handle on, assuming we've identified the risk that something might go wrong, whether there's enough additional risk that someone would break this same functionality in the future with an unintentional change to be worth keeping the regression test around long-term. As for your second question, that's an unfortunate aspect of working on a large legacy codebase. We have much better development practices now than 20 years ago when it was written, but for the present... – c32hedge Aug 25 '17 at 20:30
  • ...this is where we are. We need to continue supporting the existing software (millions of LOC) so even if we could start on a total rewrite today we'd still need to maintain tests for the existing codebase for quite some time. – c32hedge Aug 25 '17 at 20:35
  • If your regression suite consists of tests x y and z, maybe during regression test 1 you need to execute tests x and y, and during regression test 2 you need to execute tests y and z? Or in other words, when planning regression, you include tests that you have determined beforehand, would actually be valuable during this regression test. Whats valuable you can determine by what code was written and what areas can it impact. – George Aug 25 '17 at 20:51
  • Ah, I guess you're referring to what my group calls "regression rounds". While we do attempt to limit which tests we actually re-run based on risk analysis, because of the interdependencies we have taken the approach of identifying tests to not run due to low risk for a given round, rather than identifying the set of tests that should be run. I know, it's backwards, but the belief (not sure that I subscribe to it) is that this is less work than identifying every test to rerun based on 100+ fixes for a given release/round. – c32hedge Aug 25 '17 at 20:57
  • 1
    I'm afraid i have no ideas then. Until you rewrite enough parts of system or start to do smaller releases, i think you might just be stuck with monstrous regressions. Since if everything can break, they might actually be valuable in some way. Maybe rank tests? Start with high risk areas, move to lower risk areas, if there's any bug found, investigate everything around it, etc. And phase out lower tier tests as new ones which are better are added. – George Aug 25 '17 at 21:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.