Load / Performance testing tools or approach for backend

Question

I work on a team which is basically all backend. Functional and unit testing is achieved by python or perl scripts or sometimes bash scripts. But if we need to load or performance test our applications, we currently have a few scripts that generate linear load and we write some scripts that would print out some stuff to console e.g., a script will generate 1000 emails queued for a certain daemon and a couple of scripts will output process snapshot (ps), memory usage, etc. to the console and we monitor those. We can obviously write a multi-threaded script specific to each load test and something that will monitor results but that would be a very specific approach and would require us to do that for each test. Basically the reasons to use a generic tool that can be tailored for specific use.

What I am looking for and unable to find so far, is a backend tool that will help us generate load with multiple threads and can be used to collect console output and aggregate it over time to generate a tabular or a graphical result to be analyzed. There are a lot of tools available for web testing, but I am looking for something similar for backend.

We work with Perl and Python, so that is a critical part. And we are mostly working on linux or a mac. Also preferred is a open source solution.

Answer 1

Given that you're primary Linux and prefer OSS, have you tried solutions such as soap/loadUI or JMeter. I know that there's some features of soapUI that require a professional license, but, it is reasonable for most companies. One of the features with this that I believe would suit you well would be the ability to call specific tests/suites from command line. For this, you wouldn't really need to modify the code behind the tool so much as set up your scripts and call them.

I also stumbled on this answer from SO that lists a few python solutions that I've heard of, but never tried.

If you need to monitor stats from your servers, is there a specific OS that you're using? Do you already have any tools that are maybe running on your production servers that do this that you could also use on your non-production servers?

Answer 2

Visual Studio's load testing can do what you are asking. It requires a license for Visual Studio Ultimate. Even though it is geared towards web testing, it can actually execute any unit test as well.

The way it works is you create a LoadTest and that LoadTest can execute any number of scenarios and each scenario can execute any number of tests. At the Scenario level you can specify how many threads you want to use to execute the tests and how you want to execute the tests (randomly pick them, run them in order, specify percentages - i.e. Test1 runs 5% of the time, Test2 runs 95%).

If you need to generate a lot of load (more than you could generate from a single client) you can set up a controller and multiple agents to spread the tests out over multiple clients.

After the run is over it also will produce pretty graphs, charts and summary and allows you to export to excel where you can create additional pivot tables and charts if you want to. If the servers are windows servers you can also monitor performance counters very easily on them.