What's coordinated omisssion and how to properly avoid it?

Question

I recently run into the issue of the so-called "coordinated omission " when finding out the latency result of wrk doesn't look normal.

Some search leads me to How NOT to Measure Latency and Your Load Generator is Probably Lying to You - Take the Red Pill and Find Out Why, Gil Tene's mail

The argument that a closed-loop load generator doesn't reflect the real world makes sense. But both wrk and wrk2's tactics to solve the omission can't convince me, where they use a fixed maximal connections limit and compensate the CO by a mathematical method that looks like a kind of weighted average(wrk's method). I can't find any solid and formal proof that the method is correct.

I also find some discussion around some other benchmark tools. Like the discussion in K6 forum points out K6 has several execution models to solve this. And Apache JMeter provides a few suggestions.

Since the coordinated omission problem has already drawn the attention from the benchmark tool community, I really want to know if there's any serious research on this, what's the CORRECT way to avoid it, and are wrk and wrk2's tactics correct?

Answer 1

Let's imagine that you have a simple test which runs 10 requests in loop.

• 9 requests response time is 1 second
• 10th request takes 10 seconds for any reason

Total test duration would be 20 seconds, average response time will be 1.9 seconds, throughput will be 0.5 requests per second.

In my opinion ignoring or masking this "long" request is a very bad practices because such behaviour must have a valid reason.

In JMeter's world I can only think of the possible "unreliable" results due to JVM Garbage Collection which is kind of unavoidable, however it can be worked around by:

• Running JMeter in distributed mode so if one load generator gets stuck the others will still be working
• Having a spare thread pool to deliver extra load if current amount of threads is not sufficient in order to reach/maintain the target throughput via Throughput Shaping Timer and Concurrency Thread Group combination

wrk and k6 don't have distributed execution mode that's why they have to work this around by artificial manipulation on response data.