Load testing tools vs. making your own

Question

I've never needed to work with commercial load running tools before, as our test infrastructure is pretty immature here and I've used internal tools in the past. We're working on a .NET technology stack using Visual Studio TFS as our main development environment / source control, and will be needing to do load testing in the near future. We'll be load-testing the system at a number of levels - DB, client applications (e.g., what happens when X many instances of the client application hit the back-end API at once?), and so on with browser-load testing on a web application as one of the lower priority forms of load testing. I see two clear options (but am open to others):

1. Purchase a tool like LoadRunner
2. Expand the infrastructure I've built for testing to run load-testing user simulations on multiple machines at once using AWS for the machines and the simulated users (our IT department is overworked, under-funded, and bad at communication; they can't be expected to provide good support or hardware for VMs)

What are the pros and cons of buying a load-testing tool vs. developing your own for load testing? What are pitfalls I should watch for if developing my own tools? Also, what commercial options for general load testing are there besides LoadRunner, and what price ranges am I looking at?

Answer 1

From my Experience I would Say Yes to VSTS Test Edition

• I used VSTS, Continuously ran load tests for 4 Hours, Simulated Maximum of 60 requests / sec with 4 Test machines, OLTP Application
• Web Service / Biztalk / SQL Server / Replication / Service Broker based architecture
• VSTS Tests - Easily bind data to tests. This is pretty straight forward ( https://docs.microsoft.com/en-us/archive/blogs/slumley/custom-data-binding-in-web-tests)
• Learning Curve is easy (VSTS Book http://vsptqrg.codeplex.com/, Videos - https://docs.microsoft.com/en-us/archive/blogs/jmeier/performance-testing-videos-posted-to-codeplex)

We were able to simulate required load, measure performance, Latency issues in Queries, Replication, Web Service Response. VSTS has improved a lot in terms of features, support. Please take time to try it out.

Answer 2

Things to consider:

• Learning curve for Load Runner, it's quite steep from my limited exposure to it.
• Since you already use TFS have a look at the VS2010 Load Test capability. It may do all you need.
• How hard would it be to extract the necessary load / performance metrics from your existing test solution. I have seen this be the downfall of an otherwise technically excellent custom load testing solution. It was solved but created a lot of extra work and stress.
• Maintenance moving forward. How much change is there on each release, how frequent are the releases and how long would it take to upgrade test scripts to match the new release.
• Current and future cost. Don't ignore the maintenance costs of either solution.

Answer 3

My team is currently experimenting with VS2010 Load Test. So far, it seems great, but still have to try some more advanced scenario's with it. Our department recently had Load Runner dropped on us with little training. Saying that the learning curve is steep is an understatement. Also, you'd want to find some of your testers and/or dev's that have experience outside of .NET. From what we've seen, if you want to create/modify scripts yourself, it appears to need to be done in Java or C (Not all that great for our .NET shop). The licensing can also be a killer with the different types of licenses required for each protocol that tests are run in. Currently, we're experiencing a lot of difficulty getting it to interact properly with a few of our web services. Although Load Runner seems to be considered the standard, I personally think that the people using it should be specialized in it.

One thing that you will experience with any performance testing tool, it requires a lot of maintenance. Just think of the maintenance that you currently do with your test automation, and then have to do it again. It's just as fragile as any other test automation.

Other options: Visual Studio 2010 Load Test: Can be run from Visual Studio 2010 Ultimate. On a client machine, can do up to 250 concurrent users (although I would never trust 1 machine to do this). If combined with MSDN license, appears to work well with using your own machine as the controller and other physical or virtual machines as agents to run the tests. Gives a decent but not great mix of browser and connection types. At the moment, only agents can only be Windows OS.

JMeter: Open sourse. Some people swear by this solution. I've had less than limited success with it. After about 20 concurrent users on the same machine, I've noticed results being skewed. I've done tests where I ran small scripts immediately after someone else with machines with the exact same spec's and seen a 50% difference.

There's numerous other ones including open source. A lot of it will depend on what you actually require. It might be worth having your team and stakeholders sit down and decide what kind of metrics you'd each like to extrapolate from the tests. Then, you could start narrowing down what technology is going to work best for you.

I'm by no means an expert on this subject, just my 2 cents.

Answer 4

How about giving SOASTA CloudTest Lite a try? It's free, comes with real time analytics and when you're ready for scale, you can run your test from the SOASTA Global Cloud.

Take a look, maybe this is what you need: http://www.soasta.com/cloudtest/lite/ Here are a bunch of video tutorials as well: http://cloudlink.soasta.com/t5/Training-Videos/bg-p/TrainingVideos

Fred