Retries don't fix eventual consistency — var0.xyz

-->

Retries don't fix eventual consistency

2026-08-02

Retries are one of those patterns that feel so obviously correct that we rarely stop to question them. Something failed? Try again. If it still fails, wait a little longer and try again. If that doesn't work either... keep trying.

Sometimes that's exactly the right thing to do. But sometimes we're retrying where there isn't actually a failure.

When retries become a design pattern

I was once discussing a proposal for handling messages in a distributed system. Let's imagine that one service emitted a "user created" event, another emitted a "subscription created" event, and a third service needed both before it could process the payment for such subscription.

The proposed solution was straightforward. If the subscription event arrived before the user existed locally, treat it as an error. Move the message into a dead-letter queue, have someone inspect it if necessary, and retry it later.

At first glance it sounds perfectly reasonable. After all, the required data isn't there, but that framing hides an important assumption: that something has failed.

The system isn't broken

In reality, nothing may have gone wrong at all.

Distributed systems don't promise that information arrives everywhere simultaneously, or in order. They promise that, eventually, it will. That's the entire idea behind eventual consistency.

If one event arrives before another, the system isn't unavailable. It isn't in an exceptional state. It's simply behaving according to the guarantees it was designed to provide.

That's a subtle distinction, but it changes everything.

It's not a bug, it's a property

Once you recognize eventual consistency as a property instead of an error condition, the architecture starts looking very different.

Instead of treating missing information as a failure that requires intervention, you realize that this is simply yet another possible state of the system, nothing else.

Store every incoming piece of data. Each time a new one arrives, check whether all the required pieces are now present. If they are, execute the work. If they aren't, do nothing.

No retries. No dead-letter queues. No humans manually replaying messages.

The system's internal state naturally progresses as information becomes available.

Complexity disappears

One of the most satisfying things about solving the right problem is watching unrelated problems disappear along with it.

Imagine a service is unavailable for a minute. Messages pile up. Some end up in a dead-letter queue while newer messages continue flowing through the system once it recovers.

Now you have another problem to solve: replaying those older messages in the correct order. Should they jump ahead of newer events? Should processing pause until they're replayed? What if ordering actually matters?

These aren't impossible problems, but they're problems you created yourself.

If every message is simply stored until all prerequisites are satisfied, none of this orchestration exists. Events naturally become eligible for processing as the missing information arrives.

Retries not considered harmful

This doesn't mean retries are bad.

A transient network timeout? Retry it. A packet was dropped? Retry it. Distributed systems are full of short-lived failures, and retrying once is often enough to smooth over the inevitable hiccups of real infrastructure.

But it's worth asking a simple question once retries start multiplying.

If retrying once doesn't solve the problem, why would retrying five times? What changes between attempt two and attempt six? Is there actually evidence that the dependency will recover, or are we simply hoping it will?

Solve the problem you have

The bigger lesson isn't really about retries at all, it's about correctly identifying the problem before reaching for a solution. Availability problems deserve one set of tools. Eventual consistency deserves another. Treating one as the other usually leads to extra infrastructure, extra operational burden, and systems that are harder to reason about.

Sometimes the cleanest solution isn't finding a smarter retry strategy, It's realizing that there was never anything to retry in the first place.

I made a video version of this discussion if you'd rather watch it: Rethinking retries.

Thanks for reading.