What are my options for non-local environment creation?

Question

What is the industry-standard method of creating multiple environments on one physical machine?

I need to test multiple Web applications in multiple browsers on multiple OSes -- like everyone else. I know there are ways to have non-local instances of these environments; I'm wondering what is the one with the most exposure or ease.

Some thoughts I've had are:

• using a local machine with maxed out RAM and a bunch of virtual machines
• opening a few Amazon EC2 instances with the various environments, and RDP-ing into them
• exploring an existing Web app, like http://www.browserstack.com/

Thanks!

Answer 1

I've used Sauce On Demand with several clients, and quite like their services. The OSes and browsers run on their machines, and connect (in a variety of ways) to your servers. Sauce On Demand is for automated tests, driven by Selenium or WebDriver.

They also offer Scout for running manual tests through their OS/browser combos. I haven't used that, but you try it for free.

Answer 2

We work on the MS Stack, so we have servers with SCVMM on them. Using MTM and the Lab Management features allows you to spin up environments for test runs (for automation as well).

For instance and environment could be:

• Windows Server 2008 R2 web server
• Windows 7 Client machine with IE9

... or any combination of server/client OS' and browser (just happens to be one I'm working with now).

That environment can be used by the the whole team or cloned so that each person has their own set of environments.

They can get more complicated like 3-tier + client environments.

Initial configuration and setup of the architecture to support this and configuring the software and environments takes a while though. It's not a small undertaking.

Answer 3

I do not believe there is a single, industry standard, but I am sure there are many vendors who have a different opinion.

Each of the approaches you mentioned is worth exploring. If you need to support Windows, the Amazon and browserstack approaches may be the least expensive to evaluate because you will not need to purchase Windows licenses. Of course the relative long-term costs will depend upon other factors, e.g. how often and for how long you need those environments to be running and whether multiple environments need to run at the same time. (I assume they do need to run at the same time, because otherwise you would not perceive the need to max out your RAM.)

Answer 4

We use a set of servers, each hosting several Virtuals. This gives us the combinations of OSes and Browsers we need.

Answer 5

I've used VMWare in the past to maintain different environments and load them as needed, it allowed us to always maintain the right state for the machine I wanted to test with and we'd load it and then run the tests we wanted. This covered various browsers, browser versions and OSes. Microsoft's Virtual Server was an option at one point as well, though I found VMWare more stable previously, not sure what state Virtual Server is in these days.

Answer 6

You can use VMWare or Microsoft's Hyper-V or any other virtualization software. Microsoft's Lab Manager gives you even more and uses Hyper-V and there are other similar tools as well.

One thing that I would point out though is that you have an assumption that you need to include OS in your matrix of testing web sites. I would assert that a spot check would be fine for any Windows based OS and you don't need to include different OS's in your normal rotation. In my 13 years testing websites the number of issues found in different OS's is miniscule and in my opinion including those in a test matrix is a huge waste of time and resources better spent elsewhere.

One exception to that is if you are using ActiveX controls, Flash, Silverlight, etc. which could function differently on different OS's.

Another exception is non-PC platforms (Mac/Linux) where obviously you will have different browser versions which could also function differently. I have definitely seen some differences in functionality and rendering between Safari/Firefox on Mac VS PC. For your question, you will want to include those in a test matrix. Keep in mind from an automation standpoint though that while your automation will find functional issues, you still need to do a manual spot check in those browsers for any rendering issues anyways, so again trying to put a lot of effort into automating that process may not be that cost effective.