2

I recently run into the issue of the so-called "coordinated omission " when finding out the latency result of wrk doesn't look normal.

Some search leads me to How NOT to Measure Latency and Your Load Generator is Probably Lying to You - Take the Red Pill and Find Out Why, Gil Tene's mail

The argument that a closed-loop load generator doesn't reflect the real world makes sense. But both wrk and wrk2's tactics to solve the omission can't convince me, where they use a fixed maximal connections limit and compensate the CO by a mathematical method that looks like a kind of weighted average(wrk's method). I can't find any solid and formal proof that the method is correct.

I also find some discussion around some other benchmark tools. Like the discussion in K6 forum points out K6 has several execution models to solve this. And Apache JMeter provides a few suggestions.

Since the coordinated omission problem has already drawn the attention from the benchmark tool community, I really want to know if there's any serious research on this, what's the CORRECT way to avoid it, and are wrk and wrk2's tactics correct?

1 Answer 1

1

Let's imagine that you have a simple test which runs 10 requests in loop.

  • 9 requests response time is 1 second
  • 10th request takes 10 seconds for any reason

Total test duration would be 20 seconds, average response time will be 1.9 seconds, throughput will be 0.5 requests per second.

In my opinion ignoring or masking this "long" request is a very bad practices because such behaviour must have a valid reason.

In JMeter's world I can only think of the possible "unreliable" results due to JVM Garbage Collection which is kind of unavoidable, however it can be worked around by:

wrk and k6 don't have distributed execution mode that's why they have to work this around by artificial manipulation on response data.

1
  • Thanks for writing this. But AFAIK, the coordinated omission is more related to the execution model of the benchmark tool messing up the results, rather than averaged out outliers?
    – Yujia Qiao
    Dec 28, 2021 at 8:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.