The Three Idempotencies of an Agent - Venky's Substack
Venky's Substack
SubscribeSign in
The Three Idempotencies of an Agent<br>One word, three problems: one burns tokens, one double-charges the customer, and one nobody has cleanly solved.
Venky<br>Jun 30, 2026
Share
The duplicate run and the double-charge are not the same bug
Two things go wrong in production agent systems, and engineers reach for the same word to describe both.
Two things go wrong in production agent systems, and engineers reach for the same word to describe both.<br>Thanks for reading Venky's Substack! Subscribe for free to receive new posts and support my work.
Subscribe
Here’s the first. A billing webhook fires your agent, something like agent("settle the charges for order #88421", tools=[...]). The run is slow. The webhook redelivers the same event, because delivery is at-least-once and this is routine, and a second run picks up the same order while the first is still going. Now two identical agent runs are grinding through the same expensive reasoning in parallel, burning twice the tokens. If both reach the charge step, you have a race you never designed for.<br>Here’s the second. The agent charges the card. The network blips before the confirmation comes back. The job retries to be safe. Now the customer has been charged twice for the same order. This is the one that wakes someone up at 3am.<br>Both get filed under “we need idempotency,” the property that running an operation twice leaves the system in the same state as running it once. But they are not the same bug. They don’t share a cause, they don’t share a fix, and solving one does nothing for the other. I learned this the precise way: I spent a little over a thousand lines adding idempotency to an agent framework, shipped it, and then had to explain, to a colleague and (if I’m honest) to myself, which of these two problems it had solved. It was the first one. It does nothing for the second.<br>This matters now in a way it didn’t a year ago. Agents are moving from answering questions to taking actions that cost money and can’t be taken back: charging cards, placing orders, sending emails, moving inventory. The moment an agent touches a real side effect, every failure mode that distributed systems spent decades naming arrives at once, and the vocabulary hasn’t caught up. So one word gets stretched across three different failures. If you are putting an agent anywhere near something that costs money or can’t be undone, the rest of this is for you.<br>That gap between the two bugs turned out to be the most useful thing I could teach. “Idempotency for agents” isn’t one feature. It’s at least three different problems wearing one name, and most writing on the subject quietly picks one and leaves you exposed on the other two. This post is the map: what the three layers are, where each failure actually lives, and which layer I built versus which one is still an open problem.<br>Why agents break the assumption underneath idempotency
Before the layers, here’s the one fact they all descend from.<br>In a normal distributed system, when you retry a request, you retry the same request. The bytes are identical. POST /charge {amount: 4000, card: "x"} fails, so you send the exact same payload again. This is why idempotency keys work at all. You attach a key to a fixed request, the server remembers that it has seen this key, and it dedups. The request is a stable, hashable thing. You can put it in a table.<br>An agent breaks this at the root, because the agent generates the request.<br>When you retry an agent, you don’t replay a fixed payload. You re-run a non-deterministic reasoning process that produces the payload, and that process can produce a different payload the second time. The run that first decided “charge $80, full balance minus the loyalty credit” might, on the retry, may decide “charge $100, full balance.” It might call the same charge tool with subtly different arguments, or take three tool calls to get there instead of two. Same goal, different path, different requests.<br>So the foundational tool of backend idempotency, dedup on the request, has the ground pulled out from under it. There may be no stable request to dedup on. The thing you would use as a key is itself an output of the dice roll.<br>Hold onto that, because it’s the difference between the three layers. Layers 1 and 2 are mostly classic distributed-systems problems that happen to occur inside an agent. Layer 3 is the one that’s genuinely new, and it’s new for exactly this reason.<br>Layer 1: the duplicate that overlaps in time
This is the duplicate run from the opening: the same logical request fired twice while the first is still in flight. An orchestrator retries a slow run; a redelivered webhook spawns a second run for the same order. (A double-click counts too, but that’s the frontend’s job; the ones that bite are programmatic.) Both are live, so you pay for two full reasoning loops at once, and if both reach a side effect, you have a race...