Regarding Performance Analysis through Jmeter

Question

I am running a thread of 500 users for an e-application using Jmeter. My Application is deployed on 1 server and DB is deployed on another server.( Java, Tomcat Server and Oracle 11g)

My question is how to measure the appropriate results.

For instance, whenever tomcat is restarted, at Run1 results are different from Run2 since it take time for Caching and another stuff. Then in Run3 results are more faster.

Please suggest which results should be shared or what pre-conditions should be there to judge so.

Answer 1

Presumably, Runs 4, 5, 6, and 7 will level off to a steady state. The steady state numbers are what you are most interested in.

Answer 2

If you read "Continuous Delivery" by David Farley and Jez Humble, they suggest that you "burn-in" your deployment during a integration test (for some reasonable amount of time) before you run the performance tests. For example, your integration test could look for the cache to have an expected state before the assertion declares it ready.

Answer 3

Unless you're specifically investigating the impact of caching after a re-start (e.g. looking at what happens after a server outage in the middle of a working day) then analyse the runs where the cache was already populated.

If your test is reasonably long you could simply exclude the results from the first part of the test and only focus on the period of time when the system stabilised.

You could also build in a ramp-up and settle period to your test of which you always exclude when you do your analysis. I can't remember from the top of my head but I don't think there's a way to define a settle period in JMeter where results are not measured, so you have to exclude it manually after the fact.