Load testing MongoDB with JMeter

Question

I'm using JMeter 3.3 to load test MongoDB 3.4. Because I want to simulate 10k near-parallel users I'm using the distributed mode of JMeter on 35 VMs. In addition I follow many of the performance-tips for JMeter include starting it in non-gui-mode using no Listeners, increasing heap-size etc. When I add more slaves to the JMeter-Cluster I get better performance-results from MongoDB. So it is obvious that the bottleneck is still JMeter. But somehow I have a strange feeling because the 35 VMs have 8 cpu-cores each and 28 GB RAM. Some more facts on my setup:

JMeter Thread Group: 286 Threads, Ramp-Up Period 15 seconds, loop count forever, duration 300 (so in addition on 35 machines there will be 10010 threads stressing the MongoDB approximately 5 minutes).

I'm using the MongoDB Source Config. I know that it's deprecated but can't find an alternative... and a JSR223 Sampler with this groovy script:

import com.mongodb.*;
import org.apache.jmeter.protocol.mongodb.config.MongoDBHolder
import java.util.concurrent.ThreadLocalRandom;

int randomNumber = ThreadLocalRandom.current().nextInt(1, 50000001);

DB db = MongoDBHolder.getDBFromSource(„whatever“, „inka“);
DBCollection collection = db.getCollection(„profiles“);

BasicDBObject query = new BasicDBObject(„_id“, randomNumber);
DBObject result = collection.findOne(query);

SampleResult.setResponseData(result.toString().getBytes());

So finally my question: does it seem plausible to you that 35 VMs still aren't enough? I appreciate any thought!

Answer 1

I think 286 threads is kind of very low number given 8 cores and 28 gigabytes of RAM, my expectation is that you can mimic at least 10x more threads in total (> 100 000)

1. Set up monitoring of CPU, RAM, JVM heap, disk and network on a single JMeter machine. You can use JMeter PerfMon Plugin if you don't have other telemetry tool in place.
2. Amend your test plan to kick off a very large number of users, say 10 000 would be a good starting point
3. Gradually increase the load at the same time keeping an eye on the aforementioned metrics.
4. Once you see that system is running out of any of resources or start swapping or errors start occurring - check how many users were online at this moment using i.e. Active Threads Over Time listener - this is how many you can run on a single host.

See What’s the Max Number of Users You Can Test on JMeter? article for more detailed explanation of the above steps if needed.