Future-proofing Performance Test Scripts across environments for Continuous Integration - using LoadRunner and Jenkins

Question

Here is the AUT workflow across environments:

Code moves (and gets stable) from dev > QA > Stage > Prod

Data moves (and gets dirtied) from Prod > Stage > QA > dev

We have Performance Test scripts in LoadRunner (or JMeter) along with Jenkins with the corresponding plugins (HPE Automation Tools and Performance plugin for Jmeter) configured to run against Stage whenever a new build is deployed to Stage.

The run almost always fails due to the fact that scripts would need to get updated for: Front end changes & Data changes. At this point, we fix the scripts immediately and run the test, publish the results - all manually. We feel this defeats the purpose of having the pipeline in place.

I am thinking of a solution in which we can update the scripts based on QA (once the build is deployed here before it gets deployed to stage) and then update the scripts again by running it against Prod (carefully, of course) for data updates and then set it up to be triggered by Stage Job at Jenkins end. This seems like a very cumbersome process.

My question is - are there any alternative solutions to this? Do you see problems (other than touching Prod, which rarely happens in reality) in this approach?

How do you handle CI/CD/CT Using Jenkins (we can also use IBM uDeploy/urbancode as well) and LoadRunner/JMeter in a multi-environment situation like above? Are there any Best Practices documentation out there which we can refer to?

Answer 1

What kind of application is it? Which LR protocol are you using? LR HTML scripting has always been a little more prone to breaking because if your correlation is using left/right boundaries and you are looking for the names of HTML controls, the ids and names of the controls can change. You can:

1. Use a different type of correlation, for instance XML correlation.
2. Work with developers and build team to set the builds to NOT update the HTML (web page) controls every build. This was a notorious problem with early ASP.Net controls. Once setting on the build server would fix this.
3. Use a different protocol, like TruClient.
4. Use HTML protocol to write tests against the API, or the middle tier.

Some questions to ask are:

1. Do the functional tests break in your CI CD pipelines? The answer here is usually that functional tests are executed against the API (web service layer) and not through the UI. Best practice is that load tests should really be a subset of functional tests. If functional tests are executing through an API, then then can maintain the load test scripts as well.

2. Why aren't your LoadRunner tests executed at the API (web service) layer?

HTML/GUI based tests should be very limited amount of test cases. The bulk of your tests should be API based because these are less likely to break and are easier to maintain.