Apache Kafka performance #1 - linger.ms — Jack Vanlightly

This is the first in an ongoing ad-hoc series of posts on Apache Kafka performance. I have no general direction, I’ll just share interesting insights based on the performance testing I do on Apache Kafka.<br>Recently I was curious to see if there was any general performance improvement since Kafka 3.X. So I ran a suite of benchmarks with Dimster against 3.7.2 and 4.3.0. I saw two common patterns:<br>Pattern 1: Low load benchmarks showed that end-to-end latency was higher with Kafka 4.3 compared to 3.7.2. The following is a 45 minute no-record-key workload of 5000 record/s, 20 topics (120 partitions), fan-out 2 (240 consumers), full TLS, on 3 brokers each allocated 8 SMT CPUs in k8s (on my Threadripper 9980X).

Fig 1. Low load: end-to-end latency over time (p99 over 10 second intervals)

Pattern 2: On more stressful loads, 3.7.2 would show much more spiky end-to-end latency compared to 4.3. The following is for the same workload at 100K records/s (200K out).

Fig 2. High load: end-to-end latency over time (p99 over 10 second intervals). Kafka 3.7.2 showed large latency spikes.

Fig 3. High load: End-to-end latency distribution

It seemed that somewhere between 3.7.2 and now, big performance gains had occurred. Then my subconscious kicked in and reminded me that at some point in that period, the default linger.ms had been changed from 0 to 5 ms. This would correlate with the low-load end-to-end latency result.<br>Doing the math on linger.ms<br>The linger.ms producer config controls how long the producer is willing to wait before sending a non-full batch (controlled by batch.size). If a batch reaches batch.size first, it can be sent earlier. The point of linger.ms is simple: give more records a chance to accumulate into the same batch, because larger batches are more efficient than many tiny batches.<br>The important quantity is the rate “per producer, per partition” (rather than the aggregate rate). Kafka producers build batches per partition, so a producer sending 1,000 records/s to one partition has very different batching behavior from a producer sending 1,000 records/s evenly across 100 partitions.<br>A rough way to reason about it is:<br>expected records per batch ≈ 1 (first record) + linger_seconds × records_per_second_per_producer_per_partition<br>For example, linger.ms=50 with a per-producer-per-partition rate of 100, we might expect 6 records per batch.<br>This is only an approximation as it ignores arrival jitter, partition skew, batch.size config (default 16KB), compression, in-flight request limits, and broker backpressure. But it is good enough to build intuition.<br>The math and the 5K workload<br>In the 5K records/s workload, each producer was sending about 41 records/s:<br>5000 records/s / 120 producers = ~41 records/s per producer<br>That is one record every:<br>1000 ms / 41 = ~24 ms<br>This was also a no-record-key workload. With the default partitioning behavior, records from a producer tend to stick to one partition for a while before moving to another sticky partition. So, for batching purposes, the producer was usually sending roughly one record every 24 ms to its current sticky partition.<br>That makes linger.ms=5 unlikely to help. A 5 ms linger is much shorter than the ~24 ms average gap between records, so most batches still contain a single record. To reliably get more than one record into a batch, the linger would need to be on the order of the inter-arrival time (tens of milliseconds), not 5 ms.<br>So the low-load result made sense: Kafka 4.3’s default linger.ms=5 added a little extra waiting causing a higher end-to-end latency, but did not create meaningfully larger batches and its load was so low that larger batching wouldn't have helped anyway.<br>The math and the 100K workload<br>The 100K records/s workload was different. There, each producer was sending about 833 records/s:<br>100000 records/s / 120 producers = ~833 records/s per producer<br>That is one record every:<br>1000 ms / 833 = ~1.2 ms<br>At that rate, linger.ms=5 can make a real difference. A producer has time to collect several records before sending a batch. In this workload, I saw the average batch size reach about 5 KB, or roughly five 1 KB records per batch.<br>That reduced the number of small produce batches the cluster had to process. It also improved downstream efficiency for the brokers and consumers. The result was a large reduction in tail latency:<br>the 3.7.2 run, with the old default linger.ms=0, had periodic p99.9 spikes around 700 ms,

while the 4.3.0 run, with the new default linger.ms=5, had a much lower and more stable p99.9 around 8 ms.

So the benchmark was not necessarily showing a deep Kafka 3.7.2 versus 4.3.0 performance difference. A large part of the effect could be explained by one client-side default changing: linger.ms moved from 0 to 5 ms in Kafka 4.0.<br>Running a linger.ms experiment on 3.7.2 and 4.3.0<br>I decided to run a similar benchmark again, explicitly setting linger rather than using defaults. This...