Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I wanted to reach out and get some opinions of Testing hardware. Should I use VMs or a stand-alone tower? If I use VMs, what would be some recommended hardware specs for the hosting server? If a standalone physical workstation, what would you recommend in regards to processor power, ram, and gpu-wise?

Right now I have about 242 Selenium WebDriver tests spread across 11 test suites for which I run in parallel with each other. I am going to be adding at least another 360 tests in the near-future. My goal is to be able to run all the test suites simulatenously while not sacrificing the performance of each test.

Also let's say that my budget is $8,000.

Thank you all for your time and any advice you may give.

Regards, Travis

share|improve this question
    
Are you running into performance problems now? Also, am I reading your post correctly that you want to run your 11 test suites in concurrently? My gut says virtualized workstations are going to help because they can share identical blocks of memory with other VMs on the same host which will help with the number of browsers you could be opening up depending on how many tests you are running at once, but let us know where you're seeing problems and I could get into more detail. –  Daniel Jun 18 '13 at 11:55

4 Answers 4

Travis,

I'm coming into this a bit late, but here's some thoughts for you:

  • What is the total run-time of all your tests? For instance, if you can schedule your suites to run in sequence overnight and have all your results waiting for you the next morning, you may only need one system to manage testing.
  • What levels of IO do your tests require? Tests with a lot of database or other external file reads are much heavier on IO than something that simply runs through a series of actions and assertions, although any automated test will have higher IO requirements than normal running due to the need to read in the test as well as the IO of the application in test.
    • What control do you have over your testing network? If you're able to add systems (physical or virtual) to a testing subdomain without needing to work with your employer's IT department, you have a lot more flexibility to store, spin up and rename virtual systems at need.
    • In any architecture, there will be choke points. The biggest one in virtualization is the server's IO capacity, at least when hosting automation VMs. I've seen high capacity servers overwhelmed by IO requirements because there wasn't enough disk IO throughput available.

Hopefully that will help clarify your thoughts.

share|improve this answer

A couple of workstations and VMs on each of them!!! Yes, I/O will be a problem but mostly if you install a SQL database on every VM.

You may eventually want to host a staging version of your website / service on some of the VMs and use others to run tests or to run load tests.

Commodity hardware is much cheaper than high-end servers (that's one of the reasons all the cloud providers keep their costs low). So a couple of workstations will sum up to much more than a single server.

GPU should have no impact, unless you plan to crack hashes or play 3D games while your tests are running; anyway I doubt you'll find a decent GPU in any server (unless you go for some fancy designer's workstation)

We use Dell Precision workstations, $8,000 will net you 3-4 of them, depending on how many accessories like monitors you want. That's a decent test lab :)

share|improve this answer

We useSaucelabs. Budget wise the Small Team 10 CPU plan is $149 a month. There are minute limits so you would want to take total test times in consideration. Another huge plus is not having to deal with every single issue that I seemed to run into for every different OS/Browser setup.

share|improve this answer

Based on my experience, (which will vary from person to person) that automation tends to bottleneck on CPU. This lends itself to working better on actual physical hardware than on virtual hardware.

For that money I would by 13 mac minis and build a test grid. The mac mini gives you the option of running windows, mac os as well as virtualised linux, windows phone, iPhone or android clients.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.