Kafka Partitions are the wrong ordering abstraction. Keys are. | by Stéphane Derosiaux | Conduktor | Jun, 2026 | MediumSitemapOpen in appSign up<br>Sign in

Medium Logo

Get app<br>Write

Search

Sign up<br>Sign in

Conduktor

Stories from our engineering team to build the best collaborative development platform for Apache Kafka

Kafka Partitions are the wrong ordering abstraction. Keys are.

Bye “Parallel Consumer” and Why the real Kafka unit of work was never the partition.

Stéphane Derosiaux

8 min read·<br>8 hours ago

Listen

Share

Press enter or click to view image in full size

Confluent has officially deprecated the Parallel Consumer library, redirecting people to Share Consumers in Apache Kafka as the replacement , “similar functionality”. Chuck, one of our engineers at Conduktor, called this out on LinkedIn this week. I want to second him, loudly: this is the wrong call. Not the fact to deprecate this lib but to say that “Share Consumers” are the replacement. They are a completely different beast. They solve a different problem.<br>Antony Stubbs, one of the original implementers of Parallel Consumer on Confluent’s CSID team, has forked it and is actively improving it, thanks Antony: github.com/astubbs/parallel-consumer

So, what’s the deal here?<br>I spent years building event-driven and event-sourcing systems before Conduktor. The kind of systems where the partition is a deployment detail and the key: the entity id, whatever your domain calls it, is the entire universe (customer_id, order_id, etc.). I doubled down on his post:<br>Partitions are the wrong ordering abstractions TBF, keys are, and this library was “fixing” Kafka. Parallelism + Keys + Offset commit correctness is the real deal. Super useful for event-driven system and event-sourcing where the key is all that matters. You want parallelism across keys and no head-of-line blocking because one key or one external call is slow.

Almost every “we need more parallelism on this consumer” problem in an event-driven system is really a “we’re using the wrong abstraction” problem.<br>Let me walk through it.

What is Parallel Consumer?<br>Parallel Consumer is a library that wraps the vanilla Kafka consumer and lets a single consumer instance process records in parallel across configurable units of granularity:<br>KEY: the precious: per-key order. Records with the same key are processed sequentially in the application, and records with different keys can run in parallel. This is the default, and what you want.<br>UNORDERED: no ordering guarantee. Maximum parallelism. Just crunch through the records in parallel.<br>PARTITION: spin up worker threads for each partition to dispatch the work, instead of multiplexing records from N partitions sequentially<br>The challenge it’s solving is to parallelize processing in the application while keeping offset commit correctness , as Kafka does not support ack-per-message but only commit-per-batches.<br>Why is this fixing Kafka?<br>Because it parallelizes within the partitions already assigned to that consumer instance . It doesn’t steal partitions from other members of the group, and it doesn’t touch the rebalance protocol at all.<br>One instance assigned 10 partitions can process each batch in parallel: 10, 100, 1000, whatever you set as maxConcurrency, and the lib will keep ordering everything by key/timestamp.<br>You decouple your processing concurrency from your partition count.

This is the entire pitch of this lib.

Why Share Consumers are not the replacement<br>Share Consumers (KIP-932, “Queues for Kafka”) are great and production-ready since 4.2 (February 2026). They give Kafka a queue-like consumption model on top of the same topic you used to consume.<br>Each record is individually acquired by a consumer, then it is processed then acknowledge. In terms of actions: ACCEPT, RELEASE (put it back for retry), or REJECT (dead-letter it).<br>The records lifecycle becomes: Available → Acquired → Acknowledged.<br>It’s per-message acks.<br>The big difference with parallel-consumer: there is no per-partition order and no per-key order. The KIP: “The records in a share-partition can be delivered out of order to a consumer.” Can = Will.<br>For job-queue use cases (process this PDF, send this email, run this image transformation), Share Consumers are excellent. Each record is an independent task, order doesn’t matter, you want max throughput and per-record retry.<br>For event-driven systems, event sourcing? Ordering per key is the entire point.<br>For AI agents , it depends on their logic and context needs<br>If UserUpdated(user=42) arrives at offset 10 and UserDeleted(user=42) arrives at offset 50, you'd better process them in that order. Otherwise you've just resurrected a deleted user.<br>Telling people to migrate from Parallel Consumer to Share Consumers is like telling people to migrate from PostgreSQL to Redis because both store data. Sure, but the semantics and behavior is vastly different.

Partitions are an infrastructure concern. Keys are the domain concern.<br>The partition is a...