Sometimes the most resilient thing a system can do isn’t retry
Benjamin Cane
SubscribeSign in
Sometimes the most resilient thing a system can do isn’t retry
Benjamin Cane<br>Jul 17, 2026
Share
Photo by Clay Banks on Unsplash.<br>Sometimes the most resilient thing a system can do isn’t retry.<br>Most resiliency discussions focus on retries, timeouts, and circuit breakers.<br>But some of the most important resiliency patterns happen after the failure.<br>That’s where compensating transactions come in.<br>Resiliency Is About Recovery
A common mistake is thinking resiliency means preventing failures.<br>Failures are inevitable.<br>Networks fail. Services crash. Messages get lost. Requests time out.<br>True resiliency is accepting that failures will occur and having a plan to recover when they do.<br>Sometimes recovery means correcting actions that may already have occurred.<br>A Real-World Example
Have you ever swiped your card at a store, received a charge notification, only to have the terminal report an error and later discover that the charge was nowhere to be found?<br>What happened?<br>The point-of-sale terminal sent an authorization request.<br>But before it received a response, something failed: a network issue, a timeout, or a problem somewhere in the payment flow.<br>Now the terminal doesn’t know whether the transaction succeeded or failed.<br>The authorization request may have reached the card issuer and been processed, or it may not have.<br>Rather than risk leaving the customer incorrectly charged, the terminal sends a second transaction: a reversal.<br>The Compensating Transaction
That reversal is a real-world example of a compensating transaction.<br>Its purpose is simple:<br>Undo the effects of a previous action if that action completed successfully. If the original authorization never happened, nothing changes.<br>If it did happen, the reversal corrects it. Instead of determining exactly what happened, the system performs a corrective action.<br>Beyond Payments
Compensating transactions show up in many real-world distributed systems:<br>Releasing inventory after a failed order
Refunding a payment after a fulfillment failure
Canceling a reservation when verification times out
The pattern is always the same: something failed, and the system takes a corrective action.<br>Final Thoughts
When engineers think about resiliency, they often focus on preventing failures. But distributed systems fail in unexpected ways.<br>Sometimes the most resilient thing a system can do isn’t retry. It’s correct the mistake and move forward.
Originally posted on #Bengineering .<br>If this hit home, share the original with someone who needs it: Sometimes the most resilient thing a system can do isn’t retry
More from #Bengineering
Glue Services: Part Two — Data Synchronization<br>A useful companion on keeping old and new systems aligned during gradual platform change.
When modernizing legacy systems, don’t be afraid to build glue services<br>Edge translation and isolation patterns pair well with migration-focused architecture decisions.
YOLO Is a Terrible Strategy for Validating Production Changes<br>Operational changes get safer when validation and rollout discipline are part of the design.
Coding agents can’t see your architecture diagrams—fix that<br>Systems thinking improves when design context stays readable, versioned, and close to the code.
More from Latency Slayer
In-Flight Request Tracking: Lessons from Card Payments and HTTP/2<br>A deeper look at uncertainty, timeouts, and correctness when distributed requests do not complete cleanly.
Are Atomic Operations Faster and Better Than a Mutex? It Depends<br>A practical reminder that simpler-looking performance choices still need correctness tradeoff analysis.
Share
TopLatest
No posts
Ready for more?
Subscribe
© 2026 Benjamin Cane · Privacy ∙ Terms ∙ Collection notice<br>Start your SubstackGet the app<br>Substack is the home for great culture
This site requires JavaScript to run correctly. Please turn on JavaScript or unblock scripts