2

I wanted to reach out and get some opinions of Testing hardware.

Should I use VMs or a stand-alone tower? If I use VMs, what would be some recommended hardware specs for the hosting server? If a standalone physical workstation, what would you recommend in regards to processor power, RAM and GPU-wise?

Right now I have about 242 Selenium WebDriver tests spread across 11 test suites for which I run in parallel with each other. I'm going to be adding at least another 360 tests in the near-future. My goal is to be able to run all the test suites simultaneously while not sacrificing the performance of each test.

Also let's say that my budget is $8,000.

Improve this question
2
  • Are you running into performance problems now? Also, am I reading your post correctly that you want to run your 11 test suites in concurrently? My gut says virtualized workstations are going to help because they can share identical blocks of memory with other VMs on the same host which will help with the number of browsers you could be opening up depending on how many tests you are running at once, but let us know where you're seeing problems and I could get into more detail.
    – Daniel
    Commented Jun 18, 2013 at 11:55
  • Based on my experience, Performance of the VM or cloud running on commodity hardware is significantly low as compared to physical machines.
    – TechDog
    Commented Sep 21, 2017 at 11:09

5 Answers 5

Reset to default
1

Travis,

I'm coming into this a bit late, but here's some thoughts for you:

  • What is the total run-time of all your tests? For instance, if you can schedule your suites to run in sequence overnight and have all your results waiting for you the next morning, you may only need one system to manage testing.
  • What levels of IO do your tests require? Tests with a lot of database or other external file reads are much heavier on IO than something that simply runs through a series of actions and assertions, although any automated test will have higher IO requirements than normal running due to the need to read in the test as well as the IO of the application in test.
    • What control do you have over your testing network? If you're able to add systems (physical or virtual) to a testing subdomain without needing to work with your employer's IT department, you have a lot more flexibility to store, spin up and rename virtual systems at need.
    • In any architecture, there will be choke points. The biggest one in virtualization is the server's IO capacity, at least when hosting automation VMs. I've seen high capacity servers overwhelmed by IO requirements because there wasn't enough disk IO throughput available.

Hopefully that will help clarify your thoughts.

Improve this answer
1

Based on my experience, (which will vary from person to person) that automation tends to bottleneck on CPU. This lends itself to working better on actual physical hardware than on virtual hardware.

For that money I would by 13 mac minis and build a test grid. The mac mini gives you the option of running windows, mac os as well as virtualised linux, windows phone, iPhone or android clients.

Improve this answer
0

A couple of workstations and VMs on each of them!!! Yes, I/O will be a problem but mostly if you install a SQL database on every VM.

You may eventually want to host a staging version of your website / service on some of the VMs and use others to run tests or to run load tests.

Commodity hardware is much cheaper than high-end servers (that's one of the reasons all the cloud providers keep their costs low). So a couple of workstations will sum up to much more than a single server.

GPU should have no impact, unless you plan to crack hashes or play 3D games while your tests are running; anyway I doubt you'll find a decent GPU in any server (unless you go for some fancy designer's workstation)

We use Dell Precision workstations, $8,000 will net you 3-4 of them, depending on how many accessories like monitors you want. That's a decent test lab :)

Improve this answer
0

We useSaucelabs. Budget wise the Small Team 10 CPU plan is $149 a month. There are minute limits so you would want to take total test times in consideration. Another huge plus is not having to deal with every single issue that I seemed to run into for every different OS/Browser setup.

Improve this answer
0

Based on my experience, Performance of the VM or the cloud is significantly low as compared to physical machines.

My Experience:-

We have 500 selenium test cases which take around 7 hours in the cloud and same takes around 4 hours in the physical machine, Also our cloud VM has 56GB of RAM and 8 logical cores which is a higher configuration as compared to our local machine.

This performance lag is due to the fact that clouds run on commodity hardware also I heard that commodity hardware's clock speed is low

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.