Do you test using VM, User Agent, MutlipleIE etc?

Question

Since it is not possible to acquire all s/w and h/w under the sun, we often resort to using virtual machine, user agent, multiple IE etc.

I have had instance when I did not find an application functioning on a virtual machine especially multiple IE but when approached another colleague having the desired configuration, it worked well.

Given such instance I began to doubt using such simulation for testing. Do you also test using such simulation than actual device. Do you trust in testing using such simulation than using actual device?

Answer 1

Generally, virtual machines are fine for testing, as they will give the same results as real machines for 99% of functional tests. However, there are some things that should be tested on a real physical machine before you can declare the test passed:

• Drivers, and anything else in kernel space. VMs are completely real to the operating system, but sometimes the abstraction leaks. This is usually only apparent in the very low levels of the operating system, where the VM may not react exactly the same as a physical box. For these cases, at the very low level of the operating system code, you should test on physical machines and not VMs.
• CPU Architectures and specialized compilers. VMWare can simulate x86 and x64 compliant CPUs for your VM, but it can't really simulate different processor architectures. In previous projects, I've had to use physical boxes to test a driver on Itanium machines, and to see the differences when using the Intel compiler, which means compiling and running on an Intel CPU. These things are not good for virtualization, for obvious reasons.
• Performance testing and benchmarking for load and stress. In VM setups, the real physical resources are shared between all VMs on the host machine. Even when CPU cycles and RAM are partitioned, network and IO are still shared. This means that you can't really do performance testing, and certainly not benchmarking, on VMs.

The thing is, that even with the above exceptions, where I think VMs cannot be used effectively, I still use virtual machines for 98% of my work in any given year.

Answer 2

Virtualizing test environments is a great way to save on hardware and provide a centralized testing environment, but it's prone to the same types as issues as having a cluster of physical boxes if you don't have a strategy behind it.

In my company, we use a few clusters of virtual systems that all derive from a common template, so the default states are as homogeneous as they can get. For each system we create, we tend to take a snapshot of the default state to revert to if it gets dirtied by heavy use. This reduces reset time to a few seconds as opposed to upwards of an hour for re-imaging a physical machine.

In addition, this approach lends itself well to creating server environments for web site/application testing as well as desktop software testing, for running a suite of automated tests and providing automated acceptance tests. To date, we have never encountered any bugs or behavior that is related to the virtual platform; they still manage to be machine specific inside the virtual test environment.

Answer 3

We at CensoredCompanyName almost only use virtual environments. We are using massive clusters of VM Ware for each employee. That is over 1000 people.

We have very few dedicated environments and we did not encounter any problems with that. Maybe your configuration or VMWare software is flawed or the problem might be somewhere else?

So to assure you... We are using VMs for years now, without any problems from installed software or any problems in productivity.

I'm not sure however what the company uses as VM software but I'll find out for you if I can.

Cheers!

Answer 4

The truth of it is every install of an operating system is different. Your goal when testing is to make sure your environment is as controlled as possible. This is both easier and harder with virtual machines.

The advantages are that you can create a brand new machine from an image in a matter of seconds. This lets you have the application get a fresh install (so no corruption from previous tests.) Control over your environment is paramount.

The major disadvantage is the shared processor/memory state. This is where your control over the environment starts to fall apart. What if someone else is doing something intensive and you don't have the resources it says you do?

At my last job, we used VMs for our automated nightly tests and it worked out great. We knew they had the resources and a fresh environment. We also had VMs of the various systems we supported. It worked like a charm.

Welcome to the era of the cloud.

For the record, we used VMWare. I'm not endorsing it or anything. It was easy to use, but I don't have a standard of comparison, nor did I do too much of the heavy lifting (although I did write Groovy integrations for starting and stopping VMs and that was pretty easy.)