Before the Bad Write
Home
About
Contact
Dark Mode
Table of contents
Before the Bad Write<br>Here is a drill for your next DR review: remove one bad write from six weeks ago without throwing away the six weeks of legitimate writes that landed on top of it. The full checklist fails it, backups, PITR, hot replica, delayed replica, all of it. The layer that passes is the one most teams never build.
By Ruslan Tolkachev
TL;DR Backups, replicas, and point-in-time recovery are all built for loss: a failure you notice roughly when it happens. Corruption is the opposite. It’s copied by replication in milliseconds, faithfully included in every backup, and usually found weeks later, past every retention window you sized for cost. Surviving it takes provenance, knowing which writes to remove, not a restore to a good “when” that doesn’t exist.
An equipment-auction platform parses dealer feeds overnight; a nightly job writes each asset’s price. A refactor of that parser, agent-written and clean in review, declares asset IDs as int where the feed carries bigint. The largest dealer’s IDs sit past what 32 bits hold, so each one silently truncates to a smaller number, and that smaller number is some other asset’s ID. A $47,250 excavator’s price lands on a $3,100 flatbed trailer. Nothing objects: valid ID, plausible price, every row well-formed. The prices feed the valuation model, so every night’s mismatches flow into comparables, reserve suggestions, and settlement reports. It runs for forty days before a seller disputes a valuation against their own sale records.<br>The code fix is one line. The data is the problem: restore to when? The replica has the mispriced rows, and so does the delayed replica four hours behind it. Every nightly backup for forty days faithfully includes them. Binlog retention is fourteen days. And the moment before the corruption is also forty days of legitimate bids, sales, and settlements nobody will throw away. What actually worked was replaying all forty nightly runs and re-deriving everything downstream, and it was possible only because the raw feed was still there to replay.<br>Every layer in the stack was sized for a failure you notice<br>The standard disaster-recovery stack answers one failure class well: loss. A crashed instance, a dead disk, a dropped table, a region going dark. Detection is effectively instant, so a 30-day retention window feels generous, RPO and RTO become the numbers you argue about, and a hot replica reads as a safety net. Corruption breaks the assumption all of that was sized under, because nothing about it announces itself.<br>Warning A replica is not a backup against corruption. It’s a low-latency copy of the mistake. What replicas buy is high availability and read capacity: failover when the primary dies, and read-only queries served without loading it. By the time you know the write was bad, every replica already agrees with it.
And week six is the normal case, not the pessimistic one. Plausible-but-wrong data trips no alert: right shape, valid types, clean plans. Corruption is silent by construction; it surfaces from outside, a customer disputing a number or a reconciliation refusing to tie out, on the timeline of whoever notices, not the one your retention policy was built around.<br>The reflex answer is point-in-time recovery: replay the binlog or WAL to the instant before the bad write and you’ve surgically removed it. Retention gets in the way first. The archive is sized in days, because it’s a storage line nobody wants to grow (MySQL’s binlog_expire_logs_seconds, a PostgreSQL WAL archive, both usually a few days to two weeks), and week-six corruption is a month past the binlogs that held the offending statement. And even with the archive intact, a point-in-time restore is a full rewind: it removes the bad write and every legitimate write after it. Corruption found three days in sits comfortably inside every retention window, and the restore still doesn’t happen, because three days of a busy system’s writes is more than anyone will throw away to fix one column. Retention decides whether PITR is possible; the writes that landed since decide whether it’s acceptable, and that window closes within hours.<br>Note Scope this to production systems that keep taking writes. If the database is a nightly-rebuilt derived store or a dev environment, “restore to before the bad write and replay nothing” is a fine answer, and most of this post doesn’t apply to you. The trap is specific to systems where the writes after the corruption are themselves valuable and unrecoverable.
The defenses stack, they don’t substitute<br>The layers look like an escalation ladder, and in cost they are: each rung up takes more to build and run. But they don’t stand in for each other: a production system taking writes it can’t afford to lose ends up needing all four, because each...