Is there a better way to code-review JMeter test-plans?

Question

TLDR - What is the best way to review JMeter Test-Plans stored in a GitHub repo?

Currently, the code-review process that my performance-test team uses is somewhat cumbersome and of questionable effectiveness. Our team uses GitHub for version-control, and we follow the branching paradigm for check-ins, where the work done for a given story/bug/task is done in a branch of the main-line repository, then merged back into the mainline using the pull-review feature in git-hub.

This has left us with two options for reviewing changes to our test-plans: 1) Just review the raw XML in the PR diff-viewer 2) pull the un-merged branch and load each changed test-plan into a local-instance of JMeter without benefit of difference hi-lighting.

This is less than ideal - giving an effective code-review of raw XML is an issue for everyone regardless of what is being modeled. XML just isn't an easily human-readable format.

But loading the test-plans into JMeter and looking at them there is also problematic in that the information critical to any code-review, namely the exact things that changed, is lost, forcing each test-plan that changed to be reviewed in it's entirety and relying on the pull-request comments to inform the review about what changed.

As one would expect, this has led to situations where problems which could have been spotted by a code-reviewer have gone un-detected with the lamentable result of increased coding times, increased code-churn, and invalid results which have to be redone.

So my question to you, dear reader, is What is the best way to code-review JMeter Test-Plans that are stored in a git-hub repo?

Answer 1

Unfortunately JMeter .jmx files are basically XML and in case of large test plans it might be not very easy to view the diff.

You can consider using i.e.

• Ruby-JMeter which is a Ruby DSL wrapper for JMeter tests, this way the code review will look like a code-review for "normal" Ruby project

• or Taurus tool which supports definition of JMeter tests using simple YAML syntax so your test script would look like:

  scenarios:
    my-req: 
      requests:
      - url: http://blazedemo.com/  # url to hit
        method: GET  # request method (GET, POST, PUT, DELETE)
        label: homepage  # sampler label
  
        body: 'request-body-string'  # if present, will be used as body 
        body:  # generate query string based on parameters and request type
          param1: value1
          param2: value2
        body-file: path/to/file.txt  # this file contents will be used as post body
  
        upload-files:  # attach files to form (and enable multipart/form-data)
        - param: summaryReport  # form parameter name
          path: report.pdf  # path to file
          mime-type: application/pdf  # optional, Taurus will attempt to guess it automatically
  
        headers:  # local headers that override global
          Authentication: Token 1234567890
          Referer: http://taurus.blazemeter/docs
        think-time: 1s  # local think-time, overrides global
        timeout: 1s  # local timeout, overrides global
        content-encoding: utf-8  # content encoding (at JMeter's level), unset by default
        follow-redirects: true  # follow HTTP redirects
        random-source-ip: false  # use one of host IPs to send the request (chosen randomly).
                                 # False by default
  
        extract-regexp: {}  # explained below
        extract-jsonpath: {}  # explained below
        assert: []  # explained below
        jsr223: []  # explained below
  

Answer 2

Unfortunately, if you rely on JMeter GUI, the problem of code review is a bit inevitable.

However, one alternative is to create your JMeter scripts using purely Java, which allows you not only for easy code review, but also to use any Java feature( polymorphism e.g.) to enhance how you construct the script.

    //JMeter Engine
    StandardJMeterEngine jmeter = new StandardJMeterEngine();

    //JMeter initialization (properties, log levels, locale, etc)
    JMeterUtils.loadJMeterProperties("/path/to/your/jmeter/bin/jmeter.properties");
    JMeterUtils.initLogging();// you can comment this line out to see extra log messages of i.e. DEBUG level
    JMeterUtils.initLocale();

    // JMeter Test Plan, basic all u JOrphan HashTree
    HashTree testPlanTree = new HashTree();

    // HTTP Sampler
    HTTPSampler httpSampler = new HTTPSampler();
    httpSampler.setDomain("example.com");
    httpSampler.setPort(80);
    httpSampler.setPath("/");
    httpSampler.setMethod("GET");

    // Loop Controller
    LoopController loopController = new LoopController();
    loopController.setLoops(1);
    loopController.addTestElement(httpSampler);
    loopController.setFirst(true);
    loopController.initialize();

     // Thread Group
    ThreadGroup threadGroup = new ThreadGroup();
    threadGroup.setNumThreads(1);
    threadGroup.setRampUp(1);
    threadGroup.setSamplerController(loopController);

    // Test Plan
    TestPlan testPlan = new TestPlan("Create JMeter Script From Java Code");

    // Construct Test Plan from previously initialized elements
    testPlanTree.add("testPlan", testPlan);
    testPlanTree.add("loopController", loopController);
    testPlanTree.add("threadGroup", threadGroup);
    testPlanTree.add("httpSampler", httpSampler);

    // Run Test Plan
    jmeter.configure(testPlanTree);
    jmeter.run();

More details here.

Answer 3

Using jmeter-java-dsl you can implement simple test plans which are easily to code review. Here is an example:

import static org.assertj.core.api.Assertions.assertThat;
import static us.abstracta.jmeter.javadsl.JmeterDsl.*;

import java.io.IOException;
import java.time.Duration;
import org.junit.jupiter.api.Test;
import us.abstracta.jmeter.javadsl.core.TestPlanStats;

public class PerformanceTest {

  @Test
  public void testPerformance() throws IOException {
    TestPlanStats stats = testPlan(
            threadGroup(2, 10,
                    httpSampler("http://my.service")
            )
    ).run();
    assertThat(stats.overall().sampleTimePercentile99()).isLessThan(Duration.ofSeconds(5));
  }

}