Multiple Testing Environment Policy

Question

In our organization, there are 5 environments

1. Dev
2. QA -- For QA
3. UAT -- Demo and client verification
4. Stage -- Contains all approved task by the client that needs to be deployed on production 5.production

The issue is that every time we are asked to run regression on QA, UAT, AND STAGE this is causing an overhead. Can someone guide us on whether we need to completely retest or if we can run smoke or sanity tests?

Answer 1

Having multiple code environments is a common practice.

Other aspects that we aren't aware of:

• How do the environments differ by branching strategy? What is expected to be deployed on each environment? Do you have a roll-back/roll-forward policy?
• How often do you deploy to production? Are you doing CI/CD? Daily deploys? Weekly/Bi-weekly deploys?
• Do you have any test automation setup? If so, what type of tests: unit, integration, end-to-end tests?

As you can see with these additional questions, the answer really depends on policy and specifics of infrastructure setup.

Generally in my experience, no, you don't need to retest everything in each environment. You'll test most, if not everything, in the QA environment; this is an environment that contains all the new changes and is always in flux; it's where your day-to-day work occurs as a tester.

In stage/pre-production, here is where you can do testing just for deployment or integration purposes. If you have multiple, independent branches where code/features were tested in isolation, in stage, you integrate them into a deployment package. Now you need to ensure everything is integrated and works together. Some regression testing, some smoke testing, really any testing to ensure you're ready for production deployment, but it shouldn't be everything.

If you have test automation, you'll want to ensure you have that running in your pipeline and ensure you setup quality gates. If you have confidence in your test automation, it could be possible that running only the test automation will be needed on stage.

Answer 2

If you have sufficient control over the software and infrastructure and the infrastructure is comparable in each environment, I don't see a reason you would need to fully retest after the QA environment. In my experience in regulated industries, it's common to perform something like an installation qualification, verification of the integration processes, or a smoke test, but these are a very narrow subset of tests explicitly designed to confirm that all components are installed, have the correct version installed, and are configured properly.

Answer 3

As a software testing company, the process is to perform Regression Testing only on Staging environment once before code is deployed to Production. As there are multiple environments in your scenario, than we recommend your team the below:

1. Sanity Testing on Dev, QA and UAT environments.
2. Regression Testing on Stage environment
3. Smoke/Sanity Testing on Production environment.

Moreover, in case you have automation in your project than that can also be leveraged for executing Smoke/Sanity test executions.

Answer 4

It seems you are looking for help on how to approach testing in your organization's multiple environments. In general, it is best practice to run regression tests in all environments where changes have been made, including QA, UAT, and Stage. This is because each environment may have different configurations, data, and dependencies that can impact the behavior of the system.

That being said, it may be possible to minimize the overhead of testing by implementing a risk-based testing approach. This involves identifying and prioritizing the most critical functionality and testing that thoroughly, while possibly reducing the scope of testing for less critical functionality.

One way to do this is to implement smoke or sanity testing in these environments. Smoke testing is a type of testing that verifies that the most important functionality of a system is working correctly. Sanity testing is a type of testing that is used to verify that a small, specific part of a system is working as expected. Both of these approaches can help to reduce the time and effort required for testing, while still providing some level of assurance that the system is functioning correctly.

It's also worth considering whether it is necessary to test everything in every environment. For example, if certain functionality has already been thoroughly tested in QA and UAT and has not changed in Stage, it may be appropriate to skip re-testing it in Stage and focus on testing only the new or modified functionality.

Ultimately, the approach to testing in multiple environments will depend on the specific requirements and constraints of your organization and the risk tolerance for changes. It may be helpful to work with your team to identify the most effective and efficient approach to testing in your multiple environments.

Answer 5

The decision on whether to perform complete regression testing or just smoke or sanity testing depends on several factors, including:

1. The magnitude of changes: If the changes are major, then it might be necessary to perform full regression testing, while if the changes are minor, then smoke or sanity testing might suffice.
2. The risk involved: Higher risk situations require more comprehensive testing, while lower risk situations can be tested more briefly.
3. The impact of a potential failure: If the failure of a feature could have a significant impact on the end-users or the business, then full regression testing should be performed.
4. Time constraints: Full regression testing is a time-consuming process, so if there are time constraints, then smoke or sanity testing may be a better option.
5. The cost of testing: The cost of testing is a significant consideration in deciding which type of testing to perform. Full regression testing is more expensive than smoke or sanity testing.

In summary, the decision on which type of testing to perform depends on a thorough risk assessment, cost-benefit analysis, and the urgency of the testing. The objective should be to balance the need for thorough testing with the time and resource constraints of the project.