Should all test cases be in test suite?

Question

Should all test cases be reusable and in the test suite? Or are there scenarios where you would want to create a specific test case for a single user story? (or a combination of both)

e.g. a bug was solved for a specific OS. In this case I would actually like to test that specific OS and maybe one extra OS and focus mostly on the bug testing. Should you use a generic test that covers way more scenarios or could you just copy/create a test for this specific user story and use this as a throw-away functional test?

Other scenarios I could imagine are partial functionality or single time edge cases.

I tried googling for answers, if anyone could give me a source as to how this would work, that would be great.

Answer 1

Generally what you "should" do is highly dependent on your context. If you're in a regulated industry, for example, you would be required to keep a lot more documentation around the testing you perform than if you're a startup building todo apps.

Even in regulated scenarios, doing unstructured testing outside the context of your formal testing is likely acceptable and even encouraged, but you may also need to provide some form of objective evidence of the testing performed. This may not, however, have to be in the form of a test case in a test suite. I can add more detail if you're in that situation but it's not as common.

Beyond whatever the external requirements imposed by your context, there is the question of what provides value to you and your company. After all, good testing isn't about checking a box, it's about finding problems!

If the only requirement for your testing is that you gain some confidence in your bug fix, then it's really up to you how you gain that confidence. If it's a bug that you're not really worried about having regressions for, maybe you only need to do informal testing, and not embed the test in a formal suite that you run repeatedly. If you would get value out of writing a regression test to keep around, do it! You might also find some of the answers to my similar question helpful (note that I wrote that question before I learned anything about context-driven testing and would not take the approach we were taking on that project for my future projects).

For some additional reading, check out Michael Bolton's 6-part breaking the test case addiction blog series.

Answer 2

The answer will always be 'it depends' Every test case that gets recorded, needs to provide value. You do not want to clog up the suite with unnecessary tests that don't get you the information and answers you want to know about the Product. It is perfectly reasonable to have a test idea that ends up not providing much value, which you throw away, in fact even in automated scripts that perform testing, throwing away tests can be beneficial, and healthy.

For Text Based Documentation test suites, consider that any test case added is a promise to maintain, observe, and update or remove it when it is no longer needed. In a way, the documentation of testing is no different than managing a flower garden.

Answer 3

No, you don't have to write reusable test cases. But you will start to do it, when you will start to think in long-term and business-like. No, not every test case needs to be organised in a test suite.

Your example with OS shows that you do not understand software development. It depends what kind of bug was, what you found, and how your code has been designed. If it is in a part which is used by one OS, then no need to run the test on other OS'es.

I also have a feeling that you don't know the difference between bug report and test case, because your question and example are two different things.