Scope of regression testing

Question

Had a question about what other folks do for regression testing. Let me lay out what we currently have and what I think it should be:

1. We have smoke tests, which are just some minimum tests for build verification.
2. We have functional tests for features, which can be used to verify individual features coming into sprints. This tests individual features and not from the system perspective.
3. We have a QA kind of test suite which tests the product, with and without failures. More customer like, if you will.

My confusion is, what should regression include?

I am thinking of the following:

• Create component suites, out of both the set of feature tests for (2) and (3).

• Initial regression would pick component tests based on what was changed, and run it for both feature and QA suites.

• Once a week, run the full (2) and (3) suites, all components. Also use these suites for daily builds? Not sure what the difference is for you guys between whats tests run on daily jenkins build and QA automated regression testing when it comes to QA.

What do you guys for such situations?

Answer 1

Apologies in advance for a quick/simplistic answer - it's crunch time here!

My current client have automated around 80% of regression testing, the rest is done manually with an offshore team, where the basic rule of thumb is:

"If you change B, test A and C"

i.e., everything around the change - there's no point in your regression team (or the automated regression pack) repeating the functional tests you've already carried out as it doesn't add value and is just duplicating work.

It already sounds like you're on the right track - though i'm a little confused about point b. Personally, I'd run the entire regression suite for each release (unless it takes too long to run) because the objective is to detect regression issues, not unit testing issues.

Hope that makes sense?

Answer 2

When it comes time to complete a roll out we do the following. Its important to note this phase is ran on a previous version test template with no previous data. We are also in the process of pulling a portion of our regression suite to automation.

• Shallow exploratory. Test basic navigation and control functionality per page.
• Deep Exploratory. Similar to shallow but with more logic behind but not specific. This includes search results, display issues, Sorting, Etc.
• Once the bugs from this section have been resolved we move on to the actual regression.
• We currently have a google doc with each area of the system. Each area is marked with a priority to designate how much of a fine tooth comb we should be looking. First we verify the pass test to see if everything is working as intended. Then try to break anything we can on the page spending more time on high priority items. For us this would constitute the billing section of our program. -Once the focus section regression is completed we move on to our regression suite use cases that are stored on Test Lodge. They are basically specific use cases with step by step description and preconditions that simulate actual standard use cases and at times encompass multiple parts of the system. These are very beneficial as it will help cover standard cases that will be uncovered either in Beta or release which is always helpful.

Answer 3

I asked a similar question awhile back. Perhaps the answers located here may also be of assistance as you evaluate what would be the best fit for your organization. Based on the feedback I received from stackexchange, I modified our process to a mini-regression suite at every release. It tests only the showstopper level features. If I don't find bugs, I can finish it in around 4 hours testing simultaneously on 2 different platforms (OS level and internet), including our mobile app and device communications such as time clocks. I am the only tester; there is no automation for our mini-regression suite. If I find bugs, naturally it takes longer to finish the mini-regression.

Here is the link. Let me know if you have any questions. Internal Acceptance Test (not automated): How long should it take?

Answer 4

If the tests are automated, run them as early and as often as possible on the main branch. In the best case, you can pinpoint which merge had the culprit. At the very least, it will narrow the list of suspects.

That's assuming the tests don't cost you anything -- no manual steps, no extra servers and licenses beyond those you have anyway.