Huge variation in response time of same application in two different environments

Question

I have done execution in the JMeter tool in two different environments and found a huge variation in response time. For ENV1 I am getting nearly 3sec and for ENV2 it is coming < 1 sec. I checked in logs and found API logs are not taking much time only For secure session establishment(SSL Certificate) it is taking the time. Please help, will really appreciate it if it's explained in detail. (Attaching log details of both the environment) ENV1 log details:

2019-07-31 14:55:27,532 DEBUG o.a.j.u.JsseSSLManager: keyType: RSA
2019-07-31 14:55:27,532 DEBUG o.a.j.u.JsseSSLManager: Client alias: '1'
2019-07-31 14:55:27,532 DEBUG o.a.j.u.JsseSSLManager: WrappedX509Manager: getCertificateChain(1)
**2019-07-31 14:55:27,532 DEBUG o.a.j.u.JsseSSLManager: WrappedX509Manager: getPrivateKey: sun.security.rsa.RSAPrivateCrtKeyImpl@ffef9391
****2019-07-31 14:55:29,664 DEBUG o.a.h.c.s.SSLConnectionSocketFactory: Secure session established**
2019-07-31 14:55:29,664 DEBUG o.a.h.c.s.SSLConnectionSocketFactory:  negotiated protocol: TLSv1.2
2019-07-31 14:55:29,664 DEBUG o.a.h.c.s.SSLConnectionSocketFactory:  negotiated cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA
2019-07-31 14:55:29,664 DEBUG o.a.h.c.s.SSLConnectionSocketFactory:  peer principal: CN=7dkUc8BjHPYgSadbvt2Ysr, OU=0015800000jfQ9aAAE, O=OpenBanking, C=GB
2019-07-31 14:55:29,664 DEBUG o.a.h.c.s.SSLConnectionSocketFactory:  issuer principal: CN=OpenBanking Pre-Production Issuing CA, O=OpenBanking, C=GB
2019-07-31 14:55:29,664 DEBUG o.a.j.p.h.s.HTTPHC4Impl$JMeterDefaultHttpClientConnectionOperator: Connection established 10.102.45.126:65254<->3.121.54.161:443
2019-07-31 14:55:29,664 DEBUG o.a.h.i.e.MainClientExec: Executing request POST /oauth/as/token.oauth2? HTTP/1.1

While in ENV2 it is coming in a millisecond.

2019-07-31 15:52:06,371 DEBUG o.a.j.u.JsseSSLManager: keyType: RSA
2019-07-31 15:52:06,371 DEBUG o.a.j.u.JsseSSLManager: Client alias: '1'
2019-07-31 15:52:06,371 DEBUG o.a.j.u.JsseSSLManager: WrappedX509Manager: getCertificateChain(1)
**2019-07-31 15:52:06,371 DEBUG o.a.j.u.JsseSSLManager: WrappedX509Manager: getPrivateKey: sun.security.rsa.RSAPrivateCrtKeyImpl@ffef9391**
**2019-07-31 15:52:06,516 DEBUG o.a.h.c.s.SSLConnectionSocketFactory: Secure session established**
2019-07-31 15:52:06,516 DEBUG o.a.h.c.s.SSLConnectionSocketFactory:  negotiated protocol: TLSv1.2
2019-07-31 15:52:06,516 DEBUG o.a.h.c.s.SSLConnectionSocketFactory:  negotiated cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA
2019-07-31 15:52:06,517 DEBUG o.a.h.c.s.SSLConnectionSocketFactory:  peer principal: CN=7dkUc8BjHPYgSadbvt2Ysr, OU=0015800000jfQ9aAAE, O=OpenBanking, C=GB
2019-07-31 15:52:06,517 DEBUG o.a.h.c.s.SSLConnectionSocketFactory:  issuer principal: CN=OpenBanking Pre-Production Issuing CA, O=OpenBanking, C=GB
2019-07-31 15:52:06,517 DEBUG o.a.j.p.h.s.HTTPHC4Impl$JMeterDefaultHttpClientConnectionOperator: Connection established 10.102.45.126:49546<->34.246.60.233:443
2019-07-31 15:52:06,517 DEBUG o.a.h.i.e.MainClientExec: Executing request POST /oauth/as/token.oauth2 HTTP/1.1

Answer 1

Well, we can't see much more than what you've posted. Yes, there's a difference, but why we don't know.

In general, performance testing is dependent on your environment, so if you have more of them, you either need to make sure they are the same, or you should not compare results between these two environments, because then it doesn't make much sense. Even if you have one environment, if it's a shared server where other apps reside as well... what if two different teams decide to do performance testing at the same time... against the same shared server. Such a setup does exist is some companies, but if you try to interpret results from such performance testing, it might be close to nonsence.

Honestly, I've not done much performance testing, but it goes far beyond just setting up Jmeter on your machine and issuing some requests, you need to make sure the whole testing environment is under your control and that you've set it up the way you want.

Answer 2

1. To get more information on what's going on under the hood you can add the next line to system.properties file

   javax.net.debug=ssl
   

2. You can also consider disabling closing connection on each Thread Group iteration so JMeter will keep the session between iterations therefore SSL handshake will happen only once. Add the next line to user.properties file:

   httpclient.reset_state_on_thread_group_iteration=false

Additional information:

• Diagnosing TLS, SSL, and HTTPS
• Apache JMeter Properties Customization Guide
• JMeter Properties Reference