Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

At my company the QA department will verify all of the bugs fixed in builds deployed to our QA environment. We might deploy like 10 or so builds over a few weeks and they verify that the bug fixes are in fact fixed. All of this seems fine.

When we deploy the final build to staging they go and re-verify all of the bugs again. I'm not convinced that is necessary. It's the exact same code they tested in the QA environment. Maybe poke around to make sure everything installed correctly but why re-verify every single bug again? I repeat - it's the same code.

Someone help me understand.

share|improve this question
    
have ANY bugs been found to exist in the staging env and not the QA one? –  Phil Kirkham Feb 27 at 19:38
    
When you say it is the exact same code, do you mean on a binary level? –  Matt Johnson Feb 28 at 4:43

7 Answers 7

As a general rule, retesting in the staging environment is done more as an integration/sanity test than a full retest because the QA environment will likely have different code (due to other changes that aren't being pushed to the staging environment yet).

The usual considerations are:

  • The staging environment is kept as close to production as possible, and should differ only when code has been pushed to staging in preparation for deployment.
  • The QA environment has code from multiple change sets, not all of which will be targeted for deployment at any particular time (particularly when a change is part of a larger project that has to be deployed as a single chunk).
  • There is a small but not negligible chance that the difference between the two code bases will cause integration problems.
  • There is a small but not negligible chance that there are changes which are not in the change control system (This happens a lot where I work - there are data updates which happen regularly, and change control is not designed to handle things like what data is contained in which tables. Sometimes the data updates are handled as a script, but sometimes they aren't - things like bulk updates to state tax rates are generally dealt with as a manual data pump because the fields that change are different every time).
  • The continuous integration system might not handle database schema updates or stored procedure/function/user defined data type changes. These updates can be handled as SQL scripts stored in the version control system easily enough, but applying the database changes isn't something every CI system does.
  • I've yet to meet a test person who wasn't at least a little bit paranoid about bugs escaping into the wild. If there's anything different, most testers are going to want to recheck it to make sure nothing's broken.

In my experience the testing that happens in the staging environment is pretty cursory compared to what happens in the QA/Test environment, but it's never skipped.

share|improve this answer
2  
That last bullet point can't be understated. I can't tell you how many times we've pushed something from QA to staging, validated it, and then had "one minor thing" that was also requested to go to production. You hear terms like "minor", "no-impact" and "seamless" in these cases. –  corsiKa Mar 1 at 5:05
    
@corsiKa - oh, yes. I've seen one word code changes break the whole system (to be fair the word was a missing "not". In user privileges management. I freaked the whole team that day - it was late, this appeared, I knew exactly which developer did it, and I was so tired I said exactly what I was thinking at the time). –  Kate Paulk Mar 3 at 12:05

You want to know why they do this and or if it is justified.

It is difficult to answer without knowing more about the project length, product, technology and deployment environment and the acceptable defect goals.

For instance there could be a number of valid reasons. It may be the exact same code, but the deployment process may be unreliable. It may also depend how much insight or trust QA have into the latest changes and if late changes are common or discussed. It could be the QA manager uses the exercise as a final check useful for revealing any issues or side impact coincidently missed in late changes. Ie; regression not really the goal. If QA is still doing it then it has probably proved useful for some reason in your environment. It may be they just schedule full defect regression for that stage because they have no time earlier.

IME the usual approach at staging is high level smoke test to ensure all major functions are operating correctly + reverification of small number of high priority issues and recent fixes. Retesting the entire list of issues resolved including many minor issues is unusual.

share|improve this answer

Repeating all bug fix verification in stage seems like over-kill to me. One question, have you ever seen a fix fail in stage that passed in the QA environment? If this is a common problem, you should look at improving your QA environment (assuming stage is pretty close to production configuration).

One class of bug fixes that we do verify in both QA & stage: those where the underlying dependency is different. For example, my team has separate systems for authentication between pre-prod & stage. For fixes that rely on authentication, we will make sure they stay fixed on stage (which points to the production auth server).

share|improve this answer

It is kind of regression tests to me. You mention that you may have 10 builds. If you test a bug fix on the 1st build, what makes you sure it is going to be still fixed on the 10th. A regression may have been introduced within the recent builds.

And same goes for staging area. There could be some differences in the platform that make some test fails on that env. So all in all those tests are maybe not so useless.

The approach I follow usually is to automate as much as possible when my acceptance test is done on build number X. So then I can launch again that automated tests on build X+n, and once again on the staging environment. But then, it depends how easy your tests are to automate of course.

share|improve this answer

It sounds like you have a thorough manager.

There is many ways you can look at this if it gets released into production.

  • Your team discovered the defect, it was fixed and verified. You have your documentation that proves this. Somehow, after all of that it was broken again. It happens, and if the source control is weak, it can happen a lot. Your team is likely not considered to be responsible for this and the blame will likely fall on the dev team. No sweat off your back. You did your job.

  • Your team discovers the defect, it was 'fixed' and verified. It is discovered in production. Your team has 0 documentation establishing that this defect fix was verified. It's your fault. The blame comes onto you, if it's a bad defect that impacts revenue or up time, someone is going to be in a lot of trouble. And since you can't establish that this defect was retested, it is completely QA's fault.

  • Your team retests the defects. The easiest way to do this is by having the automation team set these up (I actually built a regression tool that will do this for my API). You rerun all Sev 1 and Sev 2 defects. If all are good, wonderful. If any fail, you find out why, how severe it is and whether or not it would justify a fix. Either way, the QA department has 'protected' themselves from the backlash of the defect in production.

Sadly, this is the world of QA. It's called protecting yourself. The fact that this is done so late in the release is puzzling to me. Also, if this is a process that your bosses insist on keeping than I would suggest developing a system that would make this easier to do in bulk.

share|improve this answer

Some good answers here already.

I'm going to try to take a slightly different angle: it sounds to me as if you're questioning the risk assessment of your test team.

In my experience, "it's the same code" can be a valid reason not to re-test something. It can also be that the code written by your team is exactly the same, but when deployed to a different environment, possibly with different libraries, with things running on multiple servers instead of one, talking to various third-party sites directly instead of mocked out, your deployment is flaky, etc - that it will behave differently. So, as others have pointed out, there can be various good reasons to re-test. The important question is, how do you figure out whether the risk is worth the cost?

When I decide whether I need to re-test something in a staging environment, these are some of the questions I ask to help me figure out risks:

  1. What environment differences are there, that we know of, between test and staging?
  2. Is this particular area (or areas that might affect this one) different? If we're not sure - how likely do we think it is that it'll be affected?
  3. How important is this particular functionality? What's the worst that could happen, if I don't retest and it doesn't work?
  4. Whose opinions did I ask for the previous questions? Did I miss someone important?
  5. (MOST IMPORTANT) What other tests would I not get time to run, if I do this testing?

How much visibility do you have of their decision making? How much do they have of yours?

If you think they may be making the wrong decisions, or you just don't understand why they're making those decisions, I would first suggest you try an informal chat with them to understand what affects their assessment of risk, before offering any suggestions or criticisms. It may be that it doesn't make sense because it's political, not technical (tread carefully if that might be the case). It might be that there are issues you aren't aware of. It might be that things that are very easy for you to check, aren't easy or aren't visible at all to them - and you could help out there.

share|improve this answer

One thing I've learnt over the years is that people generally have a vastly different assessment of risk. I worry a lot about high probability / low impact risk whereas my partner worries about low probability / high impact.

That assessment is inevitably coloured by the environment in which people work, and whether the organisation states it has (and genuinely does have) a 'blame-free' culture.

At one place I worked, after extensive testing, a 'final' test was failing when some new software (a web service) was deployed and running. But the same test also failed when the (new) web service was switched off and NOT running.

That suggested a test / configuration error, but it was used as a reason not to release the new software!

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.