whiletruelearn/essays/patch-paradox/
The Patch Paradox<br>I have noticed a strange side effect of AI coding agents that nobody seems to be talking about much, and it’s not the usual stuff like job displacement or code quality. It’s something closer to a math problem, and once you see it as a math problem, it’s hard to unsee.<br>Here’s the setup. Building software has never been this cheap. You can describe a feature to an AI coding agent and it writes it, wires it up, tests it, ships it. What used to take a team weeks now takes a person an afternoon. This is genuinely great and I don’t want to undersell it.<br>But the same forces that make building cheap also make attacking cheap. An AI agent can just as easily be pointed at a piece of software to probe it, poke every input field, hit every API endpoint, and look for the one place where something breaks. It doesn’t get bored, it doesn’t need coffee breaks, and it can do this at a scale no human red team ever could.<br>So now we have two agents pulling in opposite directions on the same system leading to some interesting consequences.<br>Setting Up the Two Agents<br>Let’s call the coding agent C. Its job is to write and maintain software. Let’s call the attack agent A. Its job is to study that software and find weaknesses in it.<br>Now the thing both of them are operating on is the software itself, and I want to define its size a bit more carefully than just “lines of code.” Call this S. S is really the sum of everything that exists in the system, lines of code, sure, but also the number of features, the API endpoints, the third party packages pulled in, the services talking to each other, the permissions granted, the integrations wired up. Basically, everything that could possibly go wrong is part of S.<br>The intuition here is simple. The bigger S gets, the more places A has to search. More rooms in the house means more doors and windows a burglar could try. This doesn’t mean every additional line of code is equally risky, some code is genuinely safer than other code, some architectures are cleaner than others, but on average, a bigger and more tangled system gives you more chances for something to slip through.<br>What Happens When A Finds Something<br>Say A finds a vulnerability. Naturally, C has to respond. Maybe it’s a new validation check, an extra permission gate, a new security microservice, one more field in a database schema, one more layer of code wrapped around the vulnerable part.<br>Let’s call the size of this fix P. So after the fix goes in, the new size of the system is:<br>S = S + P<br>This is where the interesting part starts. The fix genuinely reduces the risk that A just found, that’s not in question. But it also makes the system bigger. And the system being bigger means there’s more code that hasn’t been battle tested yet, more surface for the next round of scanning, and more ways this new code can quietly interact with the old code in ways nobody planned for.<br>That’s the paradox in one line: C reduces a known risk by increasing the size of the system, and increasing the size of the system may increase the unknown risk.<br>The Loop<br>Once you see it this way, the whole thing turns into a cycle rather than a one time event.<br>A finds a weakness. C patches it. The patch removes that specific weakness but adds to the total surface. The larger surface gives A more to search through next time. A searches again.<br>This is a fundamentally different kind of problem than the ones we’re used to solving in engineering. Normally when you fix things, you’re converging toward some stable, “done” state. Here, the act of fixing the problem changes the shape and size of the problem itself. It’s a moving target, and the mover is you.<br>There’s another wrinkle that makes this worse, and it’s easy to miss if you only think about vulnerabilities that exist right now. A vulnerability doesn’t have to exist at the moment code is written for it to eventually get exploited. It just has to exist, silently, until someone or something classifies it as a vulnerability. A library that was considered safe for years can turn unsafe overnight because a new technique for breaking it gets discovered, a new dependency it relies on gets compromised, or a new class of attack gets invented that nobody had thought to defend against. The code doesn’t change. The definition of “vulnerable” changes underneath it.<br>This means A isn’t just searching today’s surface for today’s known weaknesses. It’s effectively searching today’s surface for tomorrow’s weaknesses too, weaknesses that don’t have a name yet. Every piece of S that exists right now, even the parts that look completely clean under every scan available today, is a candidate for being reclassified as dangerous the moment new knowledge about attacks catches up...