Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Since it is not possible to acquire all s/w and h/w under the sun, we often resort to using virtual machine, user agent, multiple IE etc.

I have had instance when I did not find an application functioning on a virtual machine especially multiple IE but when approached another colleague having the desired configuration, it worked well.

Given such instance I began to doubt using such simulation for testing. Do you also test using such simulation than actual device. Do you trust in testing using such simulation than using actual device?

share|improve this question

4 Answers 4

up vote 6 down vote accepted

Generally, virtual machines are fine for testing, as they will give the same results as real machines for 99% of functional tests. However, there are some things that should be tested on a real physical machine before you can declare the test passed:

  • Drivers, and anything else in kernel space. VMs are completely real to the operating system, but sometimes the abstraction leaks. This is usually only apparent in the very low levels of the operating system, where the VM may not react exactly the same as a physical box. For these cases, at the very low level of the operating system code, you should test on physical machines and not VMs.
  • CPU Architectures and specialized compilers. VMWare can simulate x86 and x64 compliant CPUs for your VM, but it can't really simulate different processor architectures. In previous projects, I've had to use physical boxes to test a driver on Itanium machines, and to see the differences when using the Intel compiler, which means compiling and running on an Intel CPU. These things are not good for virtualisation, for obvious reasons.
  • Performance testing and benchmarking for load and stress. In VM setups, the real physical resources are shared between all VMs on the host machine. Even when CPU cycles and RAM are partitioned, network and IO are still shared. This means that you can't really do performance testing, and certainly not benchmarking, on VMs.

The thing is, that even with the above exceptions, where I think VMs cannot be used effectively, I still use virtual machines for 98% of my work in any given year.

share|improve this answer
    
Will remember these scenarios +1 –  Tarun May 11 '11 at 5:58

Virtualizing test environments is a great way to save on hardware and provide a centralized testing environment, but it's prone to the same types as issues as having a cluster of physical boxes if you don't have a strategy behind it. In my company we use a few clusters of virtual systems that all derive from a common template, so the default states are as homogeneous as they can get. For each system we create, we tend to take a snapshot of the default state to revert to if it gets dirtied by heavy use. This reduces reset time to a few seconds as opposed to upwards of an hour for re-imaging a physical machine. In addition, this approach lends itself well to creating server environments for web site / application testing as well as desktop software testing, for running a suite of automated tests and providing automated acceptance tests. To date, we have never encountered any bugs or behavior that is related to the virtual platform; they still manage to be machine specific inside the virtual test environment.

share|improve this answer

The truth of it is every install of an operating system is different. Your goal when testing is to make sure your environment is as controlled as possible. This is both easier and harder with virtual machines.

The advantages are that you can create a brand new machine from an image in a matter of seconds. This lets you have the application get a fresh install (so no corruption from previous tests.) Control over your environment is paramount.

The major disadvantage is the shared processor/memory state. This is where your control over the environment starts to fall apart. What if someone else is doing something intensive and you don't have the resources it says you do?

At my last job we used VMs for our automated nightly tests and it worked out great. We knew they had the resources, and a fresh environment. We also had VMs of the various systems we supported. It worked like a charm.

Welcome to the era of the cloud.

For the record, we used VMWare. I'm not endorsing it or anything. It was easy to use, but I don't have a standard of comparison, nor did I do too much of the heavy lifting (although I did write Groovy integrations for starting and stopping VMs and that was pretty easy.)

share|improve this answer
    
+1 because i agree with you. –  Hannibal May 11 '11 at 5:08

We at CensoredCompanyName almost only use virtual environments. We are using massive clusters of VM Ware for each employee. That are over 1000 people.

We have very few dedicated environments and we did not encounter any problems with that. Maybe your configuration or VMWare software is flawed or the problem might be somewhere else?

So to assure you... We are using VMs for years now, without any problems from installed software or any problems in productivity.

I'm not sure however what the company uses as VM software but I'll find out for you if I can.

Cheers!

share|improve this answer
    
so you find a defect with app on VM, do you report it a it is or also state that it was encountered on VM? –  Tarun May 11 '11 at 5:00
    
Nope. We just report it. The dev mostly work on VMs too. And we did not encounter any defects yet stating: Only on VM. Or only on Live environment. Hack, most of the build machines are VMs. :) There are really only few dedicated machines. And we are a big company in the US. –  Hannibal May 11 '11 at 5:01
    
Thanks for your comments +1, I guess your company name is censored though. –  Tarun May 11 '11 at 5:54
    
Yeah sorry 'bout that. I had to, because I just realized that I signed a stupid legal agreement.... Thanks for the +1. :) –  Hannibal May 11 '11 at 6:09
2  
I think the only reasons to mention company names are (a) the company name is recognizable (b) the info is positive (c) the use of the company name adds credibility, e.g., "This is how Fog Creek does things" or "Google does that" or "Microsoft and Amazon do this differently, but both approaches seem to work OK". I mention my previous experiences at MS by name when appropriate and positive, but not my current (small and unrecognized) employer. –  Ethel Evans May 11 '11 at 18:27

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.