What parameters to test in a performance/stress testing tool?

Question

I'm about to develop a performance testing tool for a server side app and was thinking to measure the following things:

Hardware status monitoring via SNMP, Statistics about response status and respose time

My questions are:

1. Is it viable to check for things like well-formed xml or valid json strings?
2. What other things to test to be present or to monitor/track, etc?

Answer 1

user1108631, welcome to SQA. Depending on how the terms are used and who you are talking to, performance tests can be a different beast from stress tests. For brevity, I will only refer to performance tests, but I think my answer is applicable to both.

It is hard to answer your first question without knowing more about the server. If a performance test does nothing else, it needs to measure whether (and how quickly) the server responds under load. Of course it is also important that the responses are correct. In order to keep your test resource requirements within your budget, your performance test may need to accept a lower standard of correctness than your functional tests would. For example, instead of checking everything about a response, you may need to be satisfied that the response contains the expected status code. You may need to experiment to find the right balance between keeping your performance test manageable and detecting the kinds of problems you expect to arise. If the first thing that goes wrong when your server is under load is that it starts returning poorly formed XML/JSON, you may need to check for that. Otherwise, it may not be worth the expense.

Regarding your second question, you can think about two kinds of metrics a performance should capture: capacity metrics and diagnostics. Capacity metrics describe the system's capacity, e.g. number of concurrent users, responses per second, duration between request and response, and so on. (The specifics depend on the service your server provides.) Diagnostics are measurements that help you understand why things go wrong, e.g. memory footprint, number of concurrent database connections, CPU load, and anything else you can think of that's constrained by hardware or by configuration settings.

Answer 2

In addition to the things you mentioned, I would want to know the following:

• Failure Rates as the request rate increases.
• Response times as invalid responses are submitted. (This would catch if your exception processing is taking up more time than handling a "normal" request).

Analyzing the responses for well-formed XML/JSON may be too much processing to do if you're only using one machine to test with. You wouldn't want to run the risk of skewing your results because your local CPU couldn't keep with processing the server requests/responses AND making sure that the responses are well-formed.