Most of Your Backfills Didn't Have to Happen
luminousmen
SubscribeSign in
Most of Your Backfills Didn't Have to Happen<br>Backfill (n.): an incident you scheduled yourself
luminousmen<br>Jul 28, 2026<br>∙ Paid
Share
A teammate finds a bug that’s been quietly corrupting the last month of data. A two-line fix. To clean up the history, they re-run the pipeline over the last thirty days and head to lunch.<br>They come back to doubled revenue — what a productive lunch huh?? Almost seems too good to be true!<br>Ok, what actually happened — the job wrote with INSERT in append mode, so re-running it didn’t replace the old rows, it stacked new ones on top of them. Month of data, counted twice. Every dashboard downstream now shows a company that overnight started selling twice as much as it did yesterday.<br>The bug fix was two lines. Cleaning up after the fix, however, was a full day of deleting duplicate partitions, re-running with the right write mode, and explaining to people why the whole weekend’s numbers were fiction. The job that was supposed to fix an incident instead became one.<br>That’s a backfill : re-running a pipeline over old data to rewrite partitions that already exist. Every data team does it. Most of us treat it as an unavoidable chore, and I’m not sure if it is.<br>Everybody calls backfills a “necessary evil”
Schemas change. New events show up. An audit shows up wanting the last fiscal year restated. Sometimes you genuinely have to reprocess history and nobody’s at fault. Sure, fine — some backfills are truly unavoidable.<br>What I don’t buy is extending that excuse to all of them. “Necessary evil” is the phrase we reach for so we don’t have to ask why this particular backfill exists. If a backfill is unavoidable by definition, then you don’t investigate it, and you automatically recompute, reconcile, and close the ticket which is exactly how a bug that wrecked three months of numbers gets written off as maintenance.<br>So count your last few backfills. Fewer of them were unavoidable than the phrase would have you believe.<br>Three kinds of backfills
Type A, the bug backfill<br>Something broke, you fix the code, then you re-run history with the fresh fix. The bug itself is fine, we all write them. The real issue is that it lived in production for months and nobody noticed, and when someone finally did, cleaning it up meant a hand-written script. That is the real story, not the bug: not that a bug happened, but that it stayed silent so long and that fixing it hurt.<br>Type B, the schema-change backfill<br>You added a column or changed a transform and need the new logic applied to old data. Genuinely unavoidable, because you can’t see the future, though if it’s happening every month, look closer: you’re probably materializing ad-hoc columns upstream that you could be deriving downstream, and each of those dependencies forces another backfill you never needed.<br>Type C, genuinely new data<br>You started tracking a new event and want to reconstruct history from logs where you can. Nothing failed here — you just want more history than you used to keep.<br>B and C are the real necessary evils, and in those cases the goal is to make them cheap. Type A is neither necessary nor evil. It’s a symptom.<br>Sort your team’s last five backfills into these buckets. If four are Type A, then it’s not about inevitability: almost every one could’ve been caught in an hour instead of three months, and fixed with a button instead of a script. “Necessary evil” here isn’t describing your platform, it’s describing how you let yourself off the hook for not fixing the cause.<br>Five reasons you needed that backfill
Take each Type A backfill and ask: what property was missing that made it necessary in the first place? I’ll bet the answers all come down to one of the following five.<br>1. The pipeline wasn’t idempotent. You couldn’t just re-run the job for the broken window, because a second run would double the data, break dedup, or overwrite state downstream. So you wrote a one-off script instead. A job that gives a different result on its second run is dangerous by itself, and it will get re-run, that’s the whole point of a scheduler. One extra run, and there’s the story from the top of this post (real story btw).<br>2. There was nothing to replay from. The data got transformed on the way in, written once, and forgotten. No untouched raw copy you could just re-run the fixed transform against. So instead of replaying, you’re reconstructing: piecing state back together from CDC logs, a warehouse snapshot, or a backup you’re not fully sure is complete.<br>3. The schema changed upstream and nothing warned you. A source adds a field. Turns a nullable column into a required one. Your pipeline validated none of it at ingest, so it just kept going: reading and writing, until a human finally noticed the numbers looked off.<br>💡 My favorite version of this: a source system gets migrated, or bumps a library, and quietly starts sending timestamps in local time instead of UTC. Every row...