Load testing on server returns time out

Question

First time I'm doing load testing on our server. I wrote a small program to keep posting some JSON files with httpclient.

However when threads are more than 50, the program says

INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connect.

Meanwhile, if threads are around 20 to 30, the server returns me 200(OK).

Does it mean that basic load testing is already failed?

OR still I need to analyze CPU or memory or other performance detail, then I can say the load testing is failed, if the CPU or memory are too high.

Not sure how to determine if the load testing is failed. Need some suggestion.

Answer 1

Has the test failed? That depends on the expected use of the app in Production.

If 20 threads is acceptable, then you have probably proved the app is performant. If the expected production load is greater that 50, then it isn't ready.

Firstly, you need to know what the expected load will be. Once you have that, you have something to benchmark against.

You say you get a 200 for 20-30 threads, but what I would do is time how long it takes to responsd for a single threaded call. You should in your java app be able to find some way to record how long every call takes.

Then, as you ramp up load, by running several tests you can see how the response time degrades and at what point the app fails.

Regarding the server, yes you should monitor server resources such as CPU and memory usage.

If the server is unix or Linux, then I'd suggest installing NMon. This is a fab application. This app will continually monitor server resources whilst a test is running. You can define how frequently to check resources, e.g. Every 10 seconds for a 30 minute period. Once your test has finished you can pull the file off the server. There is a spreadsheet you can download which will read an nmon result file and give you a nicely formatted output.

When gearing up a for a test, I would have nmon running before, during and after the test. That way you can see the point the server is putting under strain. E.g. For a 30 minute test, I would have nmon running for 40 minutes, which is five mins before and five mins after.

So, to sum up, my approach would be to run several tests... one thread, ten, twenty, thirty, forty and fifty. Each test to run for 30 minutes, with Nmon to run for forty minutes. Then you can analyse the server resources and from the test harness see how load affects the response times.

Answer 2

I don't think you should be re-inventing the wheel as most likely your "small program" won't be displaying charts with correlation of increasing load with decreasing throughput, how many virtual users are active at the moment, how many server hits per second you managed to produce, what are server side health metrics during test execution, etc.

So I would recommend the following approach:

1. Implement your test using a special dedicated load testing tool. There is a variety of free and open source load testing tools available in the market so you won't have to spend extra budget for that.
2. Implement your test using this load testing tool. Remember that it should represent real user as close as possible with al its stuff like Cookies, Cache, handling of images, scripts, styles, simulating AJAX requests, etc.
3. Verify that your test is doing what it is supposed to be doing by running it with 1-2 virtual users and 1-2 iterations
4. Implement monitoring of baseline OS health metrics on application under test side to report i.e. CPU, RAM and Swap usage, Disk, Network IO, your application-specific metrics if any
5. Do the same for your load generator(s)
6. Start your test with 1 user and gradually increase the load until until response time will start exceeding acceptable boundaries or errors start occurring whatever comes the first. Mind how many virtual users were online at this stage - this is your application bottleneck.

Now you should have more information to analyse, i.e. you will be able to say how many users your application can handle and where it breaks. You will be also able to state if it is caused by banal lack of resources. On the other hand if your application responds slowly but there is a headroom in terms of CPU, RAM, etc. it indicates either wrong configuration of the application (or underlying infrastructure) or implementation performance problems - in this case you will need a profiler tool to identify "heavy" methods and objects.