AI Is the Ultimate Leaky Abstraction | jonathanbeard.io Skip to content An abstraction is a promise that you won’t have to look underneath. Generated answers make that promise too, right up until the moment they break it, and the moment they break it they hand you back a bill for exactly the understanding they let you skip.

I left a promissory note at the end of the last post. I’d spent the whole thing arguing that every abstraction is really a contract (here are the inputs, here’s the behavior you’ll get back) and that it loses nothing only when the part it hides produces no effect outside that contract, that almost nothing real clears that bar because the hidden part nearly always leaks something, some side effect, some more or less than the behavior you were promised, and then I pointed at the one abstraction eating the industry right now and said it deserved more than a paragraph. This is the paragraph, grown up. If you want the machinery underneath the claim, that post is where it lives; here I just want to follow it to the end of the line.

You don’t need to write code to know this feeling; you just need to have leaned on something that mostly works. Think about the auto mode on a camera. It’s a lovely abstraction over a genuinely hard problem, the three-way balance of aperture, shutter speed, and ISO that decides how a photograph comes out. Point it at a sunny street and it nails the shot every time, and you get to never learn what those three dials actually do. Then you try to catch your kid blowing out birthday candles in a dim room, or a friend backlit against a bright window, and the camera’s little exposure model guesses wrong the way it was always eventually going to. The candles smear into blur, or your friend goes to a black silhouette in front of a perfectly exposed window. Nothing is broken. The camera did exactly what it was built to do. But the only way to rescue the shot is to drop into manual and actually understand the exposure triangle the auto mode spent years letting you ignore, and if you never built that understanding, the moment is just gone. The abstraction was free right up until the instant it wasn’t, and the price it charged was the exact knowledge it had been saving you from. Same story in the driveway, by the way: the car starts every morning and you never think about the battery, right up until the cold snap when it doesn’t, and the thing that let you not think about it is suddenly the only thing worth knowing.

This is the whole pattern, and it’s everywhere once you can see it. Computing is just where somebody finally bothered to write it down. Start where the idea started. In 2002 Joel Spolsky wrote down a “law” every working engineer already knew in their bones but had never quite named. (It’s a “law” the way Murphy’s is, an observation that keeps being right, not the kind physics or a statute can hold you to.) He called it the Law of Leaky Abstractions, and it has one clause: all non-trivial abstractions, to some degree, leak. TCP promises you a reliable stream of bytes and hides the snarl of dropped packets and retransmissions and three-way handshakes underneath. The promise holds almost all the time. Then a backhoe somewhere severs a fiber, a router upstream starts flapping, and the reliable stream stops being reliable in a way no amount of staring at your own code will explain. You open a packet capture, watch the retransmissions stack up, and there you are, debugging a layer you were assured you’d never have to see. The abstraction saved you time while it worked. It never saved you the obligation to understand what it was standing on.

That last bit is the part people forget, and it’s the heart of Spolsky’s “law” rather than a footnote to it. An abstraction doesn’t erase the complexity beneath it. It hides the complexity and changes when you pay for it. While the illusion holds you’re fast, productive, blissfully ignorant of the planner choosing your query path or the allocator deciding where your objects live. The bill comes due at the leak, and the currency the leak demands is comprehension: a working mental model of the substrate, assembled under pressure, usually at the worst possible hour. (This is the same bill I called debt a couple of posts back. Same loan, different collector.) Spolsky’s quiet warning was that as our tools climbed higher, the people maintaining the stack had to know more, not less, because every new floor was one more height the leak could fall from.1

The interface over reasoning you never see

Now set a generative model on top of that stack, and notice it’s a different kind of layer entirely. When you ask a model to write a function, summarize a contract, or propose an architecture, what comes back is an abstraction over a process of reasoning the model never shows you. The prompt goes in. Something happens inside a distribution of hundreds of billions of parameters that nobody can hold in their head. Fluent prose comes out, or code...