The Acknowledgment Gap - How Event-Driven Systems Lose Messages Without Errors - Sahan Serasinghe - Engineering Blog<br>If you have built event-driven systems for any length of time, you have probably internalized the mantra of at-least-once delivery: keep retrying until the work is done, and design everything downstream to be idempotent. It’s good advice. But there’s a subtle failure mode that hides right underneath it - one where your system faithfully reports success, commits its progress, and quietly drops work on the floor. No exception, no alert, no dead letter. Just a message that was supposed to do something, and didn’t.

I ran into this recently while debugging why a small percentage of events were mysteriously never being processed. Everything looked healthy. The producer got a 2xx. The consumer committed its offset. The dashboards were green. And yet the work never happened. This post is about that gap - the space between “the system accepted my request” and “the system actually did the work” - and why it’s one of the more dangerous places for a distributed system to lose data.

Motivation

Most write-ups about message-processing reliability focus on the well-known hazards: duplicate delivery, out-of-order messages, poison pills, consumer lag. Those are real, and there’s plenty written about them. What I found much less discussed is the case where the acknowledgment itself is a lie - where a component reports success for an operation that has only been accepted, not completed, and a second component treats that acknowledgment as permission to throw the original message away.

This is a design smell that shows up across all sorts of stacks: a queue consumer that calls an async API, a workflow engine that enqueues a job, a service that hands off to a background worker. Any time you have a handoff across an asynchronous boundary , you have the potential for this gap. So I want to walk through the anatomy of the bug from first principles, then talk about how to close it properly.

Background: two kinds of “yes”

Before we get to the bug, let’s be precise about acknowledgments, because the whole problem lives in some sloppy vocabulary.

When a system replies to your request, it can mean one of two very different things:

“I have accepted your request.” - I’ve durably recorded your intent, and I promise to try to do the work. Think HTTP 202 Accepted. The work hasn’t happened yet.

“I have completed your request.” - The work is done, and its effects are durable. Think 200 OK with a result body.

These are worlds apart, and conflating them is the root of a lot of pain. The trouble is that many APIs return the same status code for “accepted” whether or not the work will eventually succeed. A 202 (or a 204 No Content, which is even more ambiguous) tells you the request was received. It tells you nothing about whether the work will run.

Now layer on the consumer side. A huge number of event-driven systems are built on brokers that use offset-based consumer groups - Apache Kafka being the canonical example. If you want a primer, I wrote an introduction to Apache Kafka a while back. The mental model is simple:

Messages in a partition have monotonically increasing offsets.

Your consumer reads a message, does some work, and then commits (or “marks”) the offset to say “I’m done with everything up to here.”

If the consumer crashes before committing, the broker redelivers from the last committed offset. That’s what gives you at-least-once semantics.

💡 The offset commit is a promise about the past. When you commit offset N, you are telling the broker “every message up to and including N has been fully handled, and you never need to give them to me again.” If that statement isn’t actually true, you have manufactured data loss with your own hands.

Hold onto those two ideas - the ambiguous “yes” and the offset-as-promise - because the bug is what happens when they collide.

Anatomy of the pipeline

Let me describe a deliberately generic pipeline. Strip away the specific technologies and almost every async system looks like this:

The consumer’s job is to read an event and translate it into an action by calling some downstream control plane - an async job API, a workflow trigger, a task queue. The control plane accepts the request and, at some later point, a worker actually executes it.

The consumer’s loop, in pseudocode, looked roughly like this:

for {<br>msg := broker.Read()

err := dispatchJob(msg) // calls the async control plane<br>if err != nil {<br>retryWithBackoff(dispatchJob, msg)

// Always commit so we never get stuck reprocessing a bad message.<br>broker.Commit(msg)

At a glance this looks reasonable, even defensive. There’s a retry with backoff. There’s a comment explaining that we always commit to avoid getting wedged on a poison message. Someone clearly thought about failure here.

And that is exactly what makes the bug so insidious.

The bug: the acknowledgment gap

Here’s the sequence that loses data.

The...