QA Methodology: Do you retest every change on a new build or only changed code?

Question

I am leading a new QA team at my company and we are small (Myself and one other person). We are currently testing releases compiled by over 10 developers, so the ratio of developers to QA is skewed. What happens is we are constantly in a release cycle and developers are constantly checking in code that will make it into the next release. The problem with this is that releases tend to get extremely large and require extensive testing. A typical release cycle is over two months whereas we are trying to get this down to under a month.

One thing we are doing to help with this is pushing for way more automated QA testing. We are getting there, but we are definitely not in a position to reliably run automated regression on every release.

A major complaint from the development team is that the way we do our STLC is slowing down release cycles. Right now, when we get a build of the software we will do the follow:

• Confirm installation steps
• Run regression
• Test new functionality
• Confirm bug fixes
• Push to User Acceptance Testing
• Confirm no issues with clients
• Sign off on RC and ready the release for production

The development team wants us to not run this cycle every time we get a build (multiple times per release version) and instead only re-test items that specifically fail between builds and basically pick up where we left off on the previous release candidate and continue on the new one.

Since the release candidate comes in a packaged install file, I am a little hesitant to simply ignore the rest of the changes and follow their advice.

I am new to the field and I am against a team of developers with many, many more years of experience than I have in SDLC/STLC.

What do you guys think? How do you guys handle this on your teams?

Answer 1

I know this problem way too well. There's no "right" answer, unfortunately, but there are some things you can do to help with this problem.

• Dependency map - do you have a list of application features that have heavy dependencies and tend to break when changes occur in other areas? If you know changes in feature X tend to break feature Y, you know you always need to check feature Y if there are changes to feature X. On the flip side, if feature Z is nicely encapsulated and doesn't break other places, you might not need to test elsewere for fixes to feature z.
• Consider your breakage history - which areas of the system are most fragile? Which ones are most stable? If you don't feel you've got this knowledge, you should be able to retrieve it from your organization's issue tracking system or talk to project managers and others to find the information.
• Think about your overall project life cycle - it sounds as though you're working through a waterfall system where the code fixes are thrown over the wall for the testers to sign off on (apologies if this is incorrect). How can you work with the development team to improve the feedback between your team and theirs? How can they make your life easier? You make theirs easier? Both teams have the same goal: you want a good product going to the customers. My experience is that approaching devs with "How can we make this work better?" usually gets a good response. Another good conversation starter is "We could do this in our team, but it will take hours. I think if one of your folks does this, it will be a few minutes work once, and then we will only need a few minutes each time we test it." (This is really handy when asking a dev to add some hooks to make automation easier - but can also be helpful if you're looking for things like context information or a change listing to be sent with each build notification).
• Can you get a change listing? - if your build notification includes a list of files changed for that build you have a starting point of changes. This combined with the knowledge of fragile and robust areas gives you information you can use to determine whether skipping parts of the regression is going to be high or low risk (or somewhere in between).
• You are not the gatekeeper - The test team isn't really the decider on whether or not the system can be released. Usually, you're providing information to the project management or product manager about the state of the product. This mindset allows you to tell project/product management, "We can skip testing feature Y for this build, but there's a high risk of problems because changes to feature X often cause issues in feature Y" and so forth. In my experience, even if the test team is supposedly the gatekeeper, in practice they will be overridden - something that I'm all too familiar with.
• Focus on your role - The actual role of a test team is to provide information to the developers, project/product managers, and in some cases customers about the state of the application. That state information consists of the things that you've tested, the problems you've found and corrected, the problems you've found that haven't been corrected and why, and the things you haven't tested and why. This approach means that the test team no longer has sole responsibility for the quality of the application - it's shared between devs, testers, and project/program management.

This is a long way from being a complete list, but hopefully it will give you some options to work with.

Answer 2

Firstly, I've been exactly where you are and I know the pain you're going through. It's also very 'cool' at the moment for everyone to talk about 'Automation' and how it's a 'golden ticket' to Continuous Integration. Everybody wants their stuff out fast.

There are ways to tackle this, but personally, I think a lot of comes with having an understanding of the Risk in both situations. Fast or Slow.

I don't condone rushing releases, it's a terrible thing, but I do advocate testing on branched code before releases to solidify planning of regressions. You can get some really quick wins simply by working on builds and compiling the results. Never let a build through with open bugs, always note what regression testing you would do to confirm the build next time.

Now, I also don't think it's necessary to test everything every build, but only because I understand the platform I work with extremely well, and our core regression tests actually provide a good deal of coverage in the main or scarier areas of our code base (things like order placement, user management, managing payment details, security options).

Basically, I'd consider the following:

1. Test on Build (dev branch or similar)
2. Compile list of functional tests to be re-run as regressions in a RC
3. Bundle these packages together to make your release (I'm assuming multiple dev branches/builds)
4. Run through any other known changes to plan out additional functional tests (someone always wants to add something)
5. Review the RC to expand on regression testing coverage (are you covering everything you're adding?)
6. Are there any high priority or critical areas of the software NOT being covered? (too little regression coverage?!)
7. Continue with functional testing, bug fixes and regressions (as above).

There is no reason for this to be 'slow', it just takes a little forethought and planning to get good results. My rule of thumb is if I think about it twice, it's important enough to cover in testing.

The benefit of the above is it's also a process which helps support the introduction of good automation practices, where you can have all your dev branch/build functional tests turned into automated regressions and thus SAVE time in the your RC testing.

A good question to ask is "Do Releases have to be quick?" Lots of people will say yes, but what do your customers want regularly? - chances are it's bug fixes, not features. Maybe consider different types of release and how it might help create channels to focus on that deliver some things quicker, some things slower.

While this isn't an 'answer' as such, it's my opinion. When you feel more comfortable with the software you test, the regression coverage you have AND the amount of tests automated (Unit, Functional, Integration Automations) then you'll feel more comfortable making these decisions, but your role is to be cautious... way more cautious than anyone else is going to want to be. It's not an easy place to be, but it's the right place.

Good luck with adapting to their comments. You're not doing anything wrong, just think about efficiency rather than raising the risks of releasing code with testing being the only thing cut out. You can spin their requirement into something good for you and your team if you think about it right ;)

Answer 3

I have encountered exactly the same problem and there is no "right" answer, but I can share with you what I've learned:

1) Code changes in one place can cause problems in another. Unless your project is so incredibly well documented that you know exactly what a change can impact, testing the full application on releases is important.

2) Find a balance between how often you're doing full tests. If your release cycle is once per month, maybe running the full test cycle once every other week is enough. It's beneficial to both you and the developer to not wait until the end to run the full set of tests because you don't want to wait too long to discover a blocking problem, but running it every day if that's how often builds come in is probably just creating more work for yourself.

3) Automation can be really helpful. You'll probably run many of these tests dozens of times and if there's a place to automate where you can feel confident that it does as good of a job, you can save yourself a lot of time.

4) If you have to compromise (and let's face it, sometimes business needs override what we'd prefer to do), some of the best full tests to run are ones that test common processes. This is a good way to make sure you're testing the items that have the biggest impact on customers.

5) It may also help to do what they're saying as a first pass. If you can take an hour to rule out errors that are at high risk, like those that occurred on the last build, you can save yourself the time of getting to that point if it's still a problem. What the developers need to understand though is that after this pass, it is not tested, it's just ready to go through a full test. I think it does save time in the long run, but they might not understand that at first.

I hope these help. I think you're right that you need a fully successful test on any build that you plan to put into production.

Answer 4

You have many excellent answers already. Adding my 2 cents to say that your story hits close to home. Management and I have come to an agreeable compromise which is that I created "Mini regression" test scenarios for only the high traffic areas and modules of the program. The Mini Regression is performed as needed, but usually during installation testing. This will not work for all QA environments, but for ours, it works fine. If issues are found they are logged in an issues list (not a defect tracker). Management reviews the issues; some get pushed and some get forwarded to the developers. From beginning to end, if no issues are found, the Mini Regression takes about 2 hours. However if I find bugs, it could take all day depending on the bug.

My suggestion is to stay flexible! I used to get frustrated at the lack of coverage, but if management finds the current process acceptable, there is no need to change until a need arises. It is better to put creative energy into focusing on what can be done rather than on what can't be done.

Answer 5

Basically your problem is you are trying to squeeze two months of work into one, make yourself super efficient while also dealing with an inordinate amount of code, and a large group of people who are claiming that you are slowing them down. All true, and while all of the answers given will help, you will also need to use some diplomacy in being able to solve this. Like others I have been in this situation, where you are outnumbered 4 or 5 to 1 where those 4 or 5 are cranking out code that can be everywhere at once. You want to be able to minimize the risks and if your experience, and defect tracking, has shown that often one change or fix makes more then you have the feeling you need to test in more areas.

Some things you can do to minmize the impact on you (some of which need assistance from the Developers, but if they want to get the release cycles down they should want to help you out).

• Dev Test Days! Something an old company of mine did, whenever the Development Team was ready for a build they all took it, and some Test Checklists, then checked code. Yet not their own! They did their own installs and checked areas they DID NOT work on. Often this would get you past the installer issues, since you now have more eyes checking issues, and often when they found it they knew the problems and dealt with them before you, and less complaining on you since they have just done the build!
• Evolve your automation. You can't automate everything, and can't do it all at once either, so you need to add piecemeal to what you have, find one thing to you can automate quickly during a release cycle so you have that for regression.
• Understand the dependencies, having a dependency map is great (though it takes upkeep) but one thing I found that works well is talk to the Devs about what THEY think are the dependencies. Often they think its something else, especially if they are focused on one area, then talk to the person who it "might" impact then see what they say.
• Automate the builds and automate some testing, see if you can get nightly builds going and have some automation checking that build each night. That way you can be sure that some tests will be run ALL the time. Try to get your Developers to work on this as well, because what also works is whoever broke last nights build, needs to fix it.

It's hard to just cut testing, and while you are there to mitigate the risks and show what may or may not be issues, sometimes Development groups need to step up and take some responsibility in what they do. This takes a LOT of diplomacy to get them on board, understand it, and then keep it going until its part of the culture but it can be done.