Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I've never needed to work with commercial load running tools before, as our test infrastructure is pretty immature here and I've used internal tools in the past. We're working on a .NET technology stack using Visual Studio TFS as our main development environment / source control, and will be needing to do load testing in the near future. We'll be load-testing the system at a number of levels - DB, client applications (e.g., what happens when X many instances of the client application hit the back-end API at once?), and so on with browser-load testing on a web application as one of the lower priority forms of load testing. I see two clear options (but am open to others):

  1. Purchase a tool like LoadRunner
  2. Expand the infrastructure I've built for testing to run load-testing user simulations on multiple machines at once using AWS for the machines and the simulated users (our IT department is overworked, under-funded, and bad at communication; they can't be expected to provide good support or hardware for VMs)

What are the pros and cons of buying a load-testing tool vs. developing your own for load testing? What are pitfalls I should watch for if developing my own tools? Also, what commercial options for general load testing are there besides LoadRunner, and what price ranges am I looking at?

share|improve this question
1  
Others have answered your question pretty well, just one comment. When you say that you will be load testing at the client application level, just make sure that you are simulating the load from the client, i.e. hitting whatever API or communication layer that the client uses, but not actually executing the client. Navigating a client will be too slow to actually be useful when trying to generate load. –  Sam Woods Oct 31 '11 at 16:42
add comment

4 Answers

up vote 7 down vote accepted

From my Experience I would Say Yes to VSTS Test Edition

We were able to simulate required load, measure performance, Latency issues in Queries, Replication, Web Service Response. VSTT has improved a lot in terms of features, support. Please take time to try it out.

share|improve this answer
    
Thanks, Siva. This is really helpful. –  Ethel Evans Oct 28 '11 at 17:55
    
I tried one of the virtual labs on Friday, and it looks like a great fit. We already own VS 2010 Ultimate w/ an MSDN membership, so continuing with this technology makes a lot of sense. –  Ethel Evans Oct 31 '11 at 17:32
1  
Please also check on think time (time between requests), Users, Test Mix. Adding some more details on Database counters/queries which may help you for analysis. - Custom SQL Queries to identify / Capture blocking/slow running queries (DMV Queries - blogs.msdn.com/b/jimmymay/archive/2008/10/30/…) - Counters and Values for Each of them for Windows , DB, Replication issues ( blogs.technet.com/b/vipulshah/archive/2006/11/30/…) –  Siva Oct 31 '11 at 18:40
add comment

Things to consider:

  • Learning curve for Load Runner, it's quite steep from my limited exposure to it.
  • Since you already use TFS have a look at the VS2010 Load Test capability. It may do all you need.
  • How hard would it be to extract the necessary load / performance metrics from your existing test solution. I have seen this be the downfall of an otherwise technically excellent custom load testing solution. It was solved but created a lot of extra work and stress.
  • Maintenance moving forward. How much change is there on each release, how frequent are the releases and how long would it take to upgrade test scripts to match the new release.
  • Current and future cost. Don't ignore the maintenance costs of either solution.
share|improve this answer
add comment

My team is currently experimenting with VS2010 Load Test. So far, it seems great, but still have to try some more advanced scenario's with it. Our department recently had Load Runner dropped on us with little training. Saying that the learning curve is steep is an understatement. Also, you'd want to find some of your testers and/or dev's that have experience outside of .NET. From what we've seen, if you want to create/modify scripts yourself, it appears to need to be done in Java or C (Not all that great for our .NET shop). The licensing can also be a killer with the different types of licenses required for each protocol that tests are run in. Currently, we're experiencing a lot of difficulty getting it to interact properly with a few of our web services. Although Load Runner seems to be considered the standard, I personally think that the people using it should be specialized in it.

One thing that you will experience with any performance testing tool, it requires a lot of maintenance. Just think of the maintenance that you currently do with your test automation, and then have to do it again. It's just as fragile as any other test automation.

Other options: Visual Studio 2010 Load Test: Can be run from Visual Studio 2010 Ultimate. On a client machine, can do up to 250 concurrent users (although I would never trust 1 machine to do this). If combined with MSDN license, appears to work well with using your own machine as the controller and other physical or virtual machines as agents to run the tests. Gives a decent but not great mix of browser and connection types. At the moment, only agents can only be Windows OS.

JMeter: Open sourse. Some people swear by this solution. I've had less than limited success with it. After about 20 concurrent users on the same machine, I've noticed results being skewed. I've done tests where I ran small scripts immediately after someone else with machines with the exact same spec's and seen a 50% difference.

There's numerous other ones including open source. A lot of it will depend on what you actually require. It might be worth having your team and stakeholders sit down and decide what kind of metrics you'd each like to extrapolate from the tests. Then, you could start narrowing down what technology is going to work best for you.

I'm by no means an expert on this subject, just my 2 cents.

share|improve this answer
    
Thanks, this is really helpful. I took Steve's advice and tried out VS 2010 Load Test last night, using a Microsoft Virtual Lab. It looks pretty good, but I was wondering . . . do I have to run Web Tests? Or can any test / script be used as the basis of a load test? We do have an MSDN license, so that's good. Also, would it work with AWS? Our company has . . . issues . . . with getting a hold of hardware, or setting up VMs, so I need to work around those limitations. –  Ethel Evans Oct 28 '11 at 17:54
    
really not sure. These are actually the types of things that I was hoping to try out this weekend –  Lyndon Vrooman Oct 29 '11 at 1:18
    
"Our department recently had Load Runner dropped on us with little training." This is a known anti-pattern for success with any tool. Unfortunately it is the management norm with performance tools these days, without respect to vendor. Your management's behavior pretty much guarantees a low or negative ROI for LoadRunner use. –  James Pulley Oct 31 '11 at 14:30
    
In response to Ethel's comment, yes you can run other test scripts. A visual studio LoadTest allows you to execute any test set up using the Visual Studio Unit Testing framework, which means you can create any test using any .net programming language and make it as complex or simple as necessary. –  Sam Woods Oct 31 '11 at 16:39
add comment

How about giving SOASTA CloudTest Lite a try? It's free, comes with real time analytics and when you're ready for scale, you can run your test from the SOASTA Global Cloud.

Take a look, maybe this is what you need: http://www.soasta.com/cloudtest/lite/ Here are a bunch of video tutorials as well: http://cloudlink.soasta.com/t5/Training-Videos/bg-p/TrainingVideos

Fred

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.