Is it okay to mix or couple many testing environments?

Question

I am not sure if this is the correct place to ask my question. Please transfer it to the right forum if needed.

I have come across a real testing environment setup where we have SETS of environments QA (QA1, QA2, QA3), Stage(Stage1, Stage2, Stage3), Perf(Perf1). We have different versions of our product (customer facing app) in each environment. There are also back end apps and web services.

The strange thing is that QA environments share some (not all) of the backend apps/services/databases used by the Stage environment SET. As a result, if you want to test an app in QA, you need to know if its shared with Stage. Then you have to change your DB connections, endpoint urls etc. to point to the right environment. Eg. To test a QA service M which is shared with stage, you'll have to use a url for stage, not QA.

Of course, this can be frustrating initially. But, you once you know what is shared, its not so bad. But, I wonder if there can be any justification for a setup like this in a professional environment.

EDIT1 - Our revenues are in 100s of millions of dollars a year. Its good to minimize costs and be efficient, but I doubt that hardware costs are a problem for us. I have seen decoupled environments in companies with comparable net incomes. Never had any issues with urls and databases while testing in that other company.

EDIT2 - Can't we take a small subset of production data and put it into QA environment. This could help to reduce the hardware requirements/costs. Then, we could have an independent QA test environment. Edit- We will not use real customer data in test environment, we will either replace it with test logins and test information. We will only use publicly visible info.

Answer 1

I firmly believe in having a completely separate QA environment (at least the very first environment where the QAs are going to test the application); nothing shared

I understand that how difficult it is to work in shared environments. I have gone through this:

Issues:

1. You always have to remember what all links, configuration setups; you have to change before you can actually start testing (because these things might be pointing to some other environment). If you start testing, and missed any of these; start counting your valueable time as a "waste" from that point onwards.

2. Another problem (and this is very common one). You raise a bug, developer starts looking into it. He is not able to reproduce it in his local development environment. And finally he finds out that "Oh, there is an issue with the X service of Staging environment. And since, that service is pointed to QA environment, your got this issue."

So, loss of effort for all. This could have been avoided in case of independent environment.

3. Then gradually, after all such setbacks; you start remembering all the setups that you need. But at that very point, a new team member joins the team and he too gets caught in this vicious cycle

4. Sometimes, there is an issue with a particular element in your environment, but you never know about it. Why? Because you are pointing to the StageX component for last one month. And you didn't know that changes have been made to the component; also the developer who made the changes; forgot to inform you

5. Such shared environment also serve as a good excuse. Sometimes, instead of drilling down a specific issue when no reason is visible; the investigator will give a serious look and say these standard lines "I cannot find anything in the functionality. Seems like an environment issue."

So, all in all, I am completely against this idea of shared environment because I have bit the bullet.

I strongly endorse, at least (but not limited to) 1 independent QA environment which is not sharing any resources with other environments.

Benefits:

1. You don't have to remember making all the configuration related changes that you make 5 times in a day in a shared environment.

2. It limits the scope of investigation (in case of bugs and other technical issues) to this environment only.

3. It save hundreds of man hours that you might be wasting (in testing) by missing some configuration setups or because of incorrect setup.

4. Moreover, if your customer want to look at your testing environment, then you are certain that the application will work because it is running in an independent environment. And there are less chances that you will come across any surprises during demo or trial as there are no external dependencies.

Answer 2

While it's great to have completely isolated environments, that also has issues (e.g. getting good data sets).

In my experience, companies are going to do what makes sense for their problems. My current employer has 4 environments, with prod and an integration environment that are customer facing. Our dev and staging infrastructures generally have their own backends, but for somethings, staging shares data with the integration environment. And in some of the lower environments, third-party integrations aren't fully functional (e.g. our dev environment doesn't talk to one of our major third-party integrations since they would basically charges us double to have a second environment up and running).

As a workaround in our tests, I try to go and gather environment info and include it in the test artifacts.

Answer 3

Ideally, testing environments should be independent of each other; otherwise, coupled testing environments would introduce extra time and effort to isolate a variable.

What you described in your question is similar to having a resource farm.

Having a resource farm has its own advantages and disadvantages:

• A resource farm offers cost saving: instead of having a database for every testing environment, a database is to be shared. This applies to many other shared resources as well.
• The disadvantage is: you will need some kind of resource management tool that scans shared resources upon request and directs you to resources that are available.

EDIT1 - Our revenues are in 100s of millions of dollars a year. Its good to minimize costs and be efficient, but I doubt that hardware costs are a problem for us. I have seen decoupled environments in companies with comparable net incomes. Never had any issues with urls and databases while testing in that other company.

• Having revenues of 100s of millions of dollars does not always mean you get to spend more money;

• If you use revenue as the argument to request more resources from the management; they can equally argue "We are making 100s of millions of dollars by having shared testing environments, why bother spending more on building more testing environments?"

• If you use other companies have decoupled testing environments as the argument; it is unlikely to convince the management either.

• Unless you can offer them tangible evidence, for example, how many man-hours are needlessly spent on waiting for the shared resource to become available; how many false-positives are caused by sharing resources and how many man-hours are spent on chasing and closing those false-positives.

EDIT2 - Can't we take a small subset of production data and put it into QA environment. This could help to reduce the hardware requirements/costs. Then, we could have an independent QA test environment.

• I agree with you in principle, but you have to consider "Are you allowed to do it" first. Production data are taken from real business customers, are those customers willing to share their information with testers? If the business you are in is sensitive in nature, the management will be very unwilling to share real data with the testing team. Last time, we requested real data to be used in a testing environment, the management declined our request based on confidentiality.