We have unit and integrations tests in our project and want to work test driven. One questions that raise up is this: When a ticket is created because somebody found a bug, should we create a small test case just for that bug, or should we include this case in the general tests?
5 Answers
Welcome to SQA, Tim.
I do not create a new test case for every bug. I believe testing is about managing risk. If I believe the risk of a a bug reappearing is low, I will test the bug when the developer says it is fixed, but I will not re-test it for every subsequent release. Judging risk is subjective, and sometimes my judgement is wrong, but I think this way of testing is a better use of resources than testing for every bug that has ever appeared.
Other organizations may manage risk differently because of the nature of their business, their regulatory environment, or the culture of the company.
-
Isn't the fact, that this bug existed in the first place, a good indicator for a fragile feature in most cases? Commented Mar 23, 2012 at 9:36
-
If by fragile you mean likely to have more bugs, I do not think so. But I think that is a different issue than whether a fixed bug is likely to break again some time in the future.– user246Commented Mar 23, 2012 at 12:58
Creating a test based on a bug is a good way to drive the development and provide an acceptable pass condition for the fix.
While i agree that smaller bugs that are hard to re-create probably aren't worth building a test around, there may be some merit in seeing how often it does actually pop up and the test does fail.
I believe the answer is "it depends". If you're talking about a unit test, why not? These should be the bulk of your tests anyway, and usually run very fast. But it may not be worth the time or money.
If you're talking about functional tests, probably not. But again, it depends. Was the bug a cross-browser or OS issue? Was it a logical bug or typo? Will the test give you a satisfactory level of confidence that it wouldn't happen again? What is the business impact if the bug happens again? How much does it cost the company to have you write a test for a particular bug? Is there sufficient ROI or risk mitigation to make it worthwhile?
These are the types of questions that you need to prioritize before you can really determine if a particular bug should have a corresponding test, and the business needs usually take precedence over all else.
-
Exactly. If you always re-test every bug that has ever been fixed, do you necessarily get better quality? It may be better to spend that time testing in other ways. Or to spend less time testing.– user246Commented Mar 23, 2012 at 16:12
Just like with everything else in IT - it depends. Depends on whether the bug can be easily reproduced, it depends on how severe this bug is, depends on the chance of the bug reappearing again somewhere in the future. Someone said before me that it's all about managing software testing risk and I agree with it. While it may seem to be tempting to cover EVERYTHING with tests, it is a waste of time and resources, in the long run, it would be writing tests for the sake of writing tests.
I'm not sure what the difference is between a "small test" and "general tests". Anyway, we add tests based on bugs to our main test suite, with some exceptions. One of the exceptions is the test not being a "small test", some bugs are difficult and time consuming to replicate or verify. Another exception is "not our bug", meaning a bug in an adjusting module that we are not responsible for (unless it's really trivial to be tested).
