Load testing APIs & web services resource?

Question

I'm looking for a good, in-depth resource for load testing REST APIs and web services that goes well beyond "beginner load testing". I'm wanting a technology-agnostic resource that focuses on the goals and techniques, and maybe suggests tools to accomplish those techniques - but not something that is tied to a particular stack or toolset. If the resource includes related testing, like performance and scalability for web apps & web services, that's a bonus.

To be clear: I'm not looking for articles on how to load test, tutorials, or tool recommendations; we're already doing basic load testing and getting value from it. I want something in-depth to help identify holes in our current approach (if any) and improve what we are already doing, plus wanting to round out my own knowledge on the topic. A quick look on Amazon isn't getting me anything that sounds promising, so I thought I'd turn to the experts.

Any recommendations? Is there a clear "best in class" resource for this?

Answer 1

I can suggest a few resources based on the Microsoft stack, i.e. using Visual Studio load testing tool:

http://vsptqrg.codeplex.com/

You aren't really tied to any of the tools, and you can do it with free tools as well, for example with Fiddler.

The general approach would be:

a) gather the usage profile - i.e. how many times each api is hit in the unit of time (an hour for ex) b) write tests that simulate hitting each API c) set a goal for your load test

• if you want to find out what the bottleneck is, try an incremental load (or if you want to be really fancy, a binary search goal)
• if you want to do stress testing or to analyze response time under load, set a constant load

d) configure your load test to run each individual test in such a way that you mimic the real usage profile; this is where it gets tricky and you many need to use a $$$ tool - because a single computer might not be able to generate enough traffic you may need to use several coordinated load agents e) run the test and monitor the systems under test (there's a whole theory on what to monitor); professional tools like Visual Studio will choose a base set with what to monitor for you - they will also suggest thresholds

Full Disclosure: I work for Microsoft

Answer 2

As per your query,

"I'm wanting a technology-agnostic resource that focuses on the goals and techniques, and maybe suggests tools to accomplish those techniques - but not something that is tied to a particular stack or toolset."

..the first thing that came to my mind was the way Netflix approaches their testing. Specifically, I thought of the Chao Monkey and some of their approaches to testing their web services. They use many different technologies to test their services and they tend to open source a lot of it. You need to have some infrastructure to take advantage of it (since they are using the AWS, you would need to adapt it to your needs)

You asked "Any recommendations? Is there a clear "best in class" resource for this?": I would say that Netflix is best in class and suggest looking at the whole Netflix infrastructure (start with the blog and and follow links to Github, etc). Since they are responsible for over 30% of all the traffic on the Internet, they know how to deal with Web services from a resiliency, performance, load, functionality, fail-over, load balancing, etc.

Your mileage might vary since it depends on the project you're apply this to. However, I think the goals, techniques, and tools to accomplish your testing can be found on their developer and technical blog pages.