The Top-Down Bet Needs A Bottom-Up Audit - by Nitin Khola

SubscribeSign in

The Top-Down Bet Needs A Bottom-Up Audit<br>I still think abstraction is where human advantage moves in agentic coding. But, I also don't think abstraction gets to ignore the machine.

Nitin Khola<br>Jul 21, 2026

Share

My bet is still on top-down AI-assisted software engineering. Durable human advantage moves up the abstraction tree while agents absorb the leaves. I’ve argued the two halves of that position already in the last two posts, and I’ve been living it for a year, watching agents take over code I used to be proud of writing.<br>A bet deserves an audit, though. So this finale in a 3-part series is a status check rather than a memoir: what the trend data shows as of mid-2026, why the loudest skeptics are conceding ground, and where the bet could still lose.<br>Thanks for reading Khola.Blog: Post-Human Engineering! Subscribe for free to receive new posts and support my work.

Subscribe

The Leaf Nodes Are Being Absorbed, On Schedule

The claim that agents will keep improving at implementation work doesn’t need my intuition. It has a measured curve.<br>In October 2023, the original SWE-bench paper reported the best model of the day resolving 1.96 percent of real GitHub issues. OpenAI’s cleaned-up SWE-bench Verified launched in August 2024 with GPT-4o at 33.2 percent. Claude 3.7 Sonnet reached 62.3 percent in February 2025. GPT-5 claimed 74.9 percent that August. By mid-2026 the benchmark is close to saturated, which is why the labs have mostly stopped bragging about it.<br>METR measures the same slope differently: the length of task a model can finish at a 50 percent success rate. Their March 2025 paper found that horizon doubling roughly every seven months; their January 2026 update put the frontier at over five hours of human-equivalent work and the doubling time closer to three months. Whatever else is true, “the leaf nodes will improve” stopped being an opinion sometime in 2024. It’s the most consistently measured trend in software. Believe the slope.<br>Two numbers complicate the slope, and they belong in the same paragraph as the cheerful ones. When Scale built SWE-bench Pro (September 2025) to resist contamination, with larger multi-file tasks from unfamiliar repos, the same class of frontier models dropped to roughly 23 percent. And practitioners are voting with their anxiety: in the 2025 Stack Overflow survey, 84 percent of developers use or plan to use AI tools while only about 3 percent highly trust the output and nearly half actively distrust it. Adoption tracks the curve. Trust tracks the gap.<br>Read together, the numbers say the absorption is real and fast where work resembles the training distribution, and thins where tasks get larger, stranger, and less checkable. Which raises the only question that matters for the bet: what, exactly, makes work checkable?<br>Why the Skeptics Are Turning

The engine under the curve is not a secret. OpenAI’s o1 described it in late 2024 and DeepSeek-R1 demonstrated it in the open in January 2025: reinforcement learning against answers a machine can check. Code is the ideal substrate for that loop, because the compiler and the failing test are free verifiers, available at whatever scale training needs.<br>That mechanism explains the most interesting sociological fact in the field. The skeptics are conceding. And they’re all conceding the same narrow thing. François Chollet spent years as the discipline’s most rigorous LLM skeptic, and when o3 cracked his ARC-AGI benchmark in December 2024 he called it a genuine breakthrough while founding a lab premised on a different road to intelligence. By December 2025 he was crediting “remarkable progress on LLM-driven refinement loops” in commercial frontier models. Andrej Karpathy’s sober middle position is the same concession from the other direction: agents genuinely handle the routine layer, and still bloat the codebase the moment the architecture gets custom. Nobody converted to scaling faith. They conceded that refinement loops against checkable feedback work, and that coding is where feedback is cheapest. Cheap feedback compounds.<br>That’s the audit’s core finding. The trend is steepest exactly where verification is cheap, and the tree is really a gradient of verification cost. A leaf has a verifier the training loop can afford: it compiles, or the test goes red. One level up, the verifier gets expensive, because a module boundary proves itself over months of change traffic. At the root, the verifier is an incident review, or the market. So the bet, restated with the precision it lacked: agents keep absorbing everything with a cheap verifier, corner cases included, because a corner case is just a test nobody wrote yet. That half I’d sign today. What stays human is the layer where feedback arrives late and entangled, and no training signal exists to climb it.<br>Honesty about the evidence: I looked for a controlled study isolating how much agent success depends on...