You can just choose how many bugs you want now | Read the Tea Leaves

Read the Tea Leaves Software and other dark arts, by Nolan Lawson

Home

Apps

Code

Talks

About

" Using AI to write better code more slowly

16<br>Aug

You can just choose how many bugs you want now

Posted August 16, 2026 by Nolan Lawson in software engineering. Tagged: AI. Leave a Comment

There’s a bizarre aspect of AI coding that I’ve been trying to put my finger on, and I think it’s this: you can basically just decide how many bugs you want your software to have now.

We discovered this first with security, because of course security bugs are the most non-negotiable ones. But I think once the vulnpocalypse is over, we’ll start to turn our attention to other types of bugs: correctness, performance, accessibility, reliability, etc.

Some of us are already doing this. For example, I find myself spending a lot of time these days in code review, using tools like my triple-agent code review skill as well as Geoffrey Litt’s explain-diff skill.

My experience is that, in a complex system, you can basically find as many bugs as you ask the agents for. If you get tired of tackling bugs in the PR itself, have no fear: the agent will also find plenty of preexisting bugs for you to spend time on. The question is just when you want to stop and call it "done."

Of course the bugs are not free to fix. There are still many tradeoffs to consider: lines-of-code versus likelihood that the bug will actually occur, the risk of introducing new bugs in a complex solution, the cost of making the code harder to understand for future reviewers or agents, etc. But the finding of the bugs has become nearly free, and AI agents are also capable of finding very subtle, intricate bugs that otherwise could have flown under the radar for years. What we do with this situation is the interesting question.

As many have noted, it doesn’t seem like the overall polish of software has increased since AI coding became a thing. If anything, there is just more junk and shovelware out there, of dubious quality. I think this demonstrates that, although our ability to find new bugs has skyrocketed, our overall tolerance for bugs has not changed. There are still plenty of winds blowing in the opposite direction:

The preventable problem paradox: if an incident occurs and you swoop in to fix it, you’re a hero. If you prevent the problem from ever occurring in the first place, then nobody knows you did anything.

Related: the pressure inside many software orgs is to keep shipping visible results, not to fine-tune something that already "works." With AI coding this is magnified: management often assumes that 10x productivity means 10x more visible features and apps.

Laziness: one of the classic virtues of a programmer, this time working against us. I find myself mentally exhausted after slogging through the umpteenth AI-generated bug report, which requires me to carefully think through intricate aspects of the system and weigh the pros and cons of fixing it. I imagine many of my peers in the industry have just tuned out AI code reviews or only focus on the most critical findings.

Avoiding epicycles

There are a few ways we can approach this problem, though, that don’t require unending toil. One way is to set up the agent on a loop, e.g. "do a code review, fix all critical/high/medium issues, then repeat." I find this can work, but it has a tendency to create lots of epicycles.

If you’re not familiar with the concept: in the pre-Copernican1 model of the solar system, ancient astronomers "fixed" miscalculations in the planets’ orbits by simply adding more circles to their movement. This improved the accuracy of the predictions, but at the cost of making the overall model more complicated. Obviously just saying "the earth moves around the sun" greatly simplifies the whole thing, but first you need the insight to make this simplification possible.

I’ve found that AI agents are pretty bad at such dramatic simplifications (in other words, "LLMs can’t jump"). They will happily build one epicycle per bug until the code is a spaghetti mess. So a valuable part of AI code review is still to ask questions like "How can we make this simpler?" and "Is there a fundamental flaw with the codebase that we should fix before we tackle this class of bugs?"

Another technique that works well is to have good tests. (Easier said than done!) For example, when I was playing around with vibe coding the W3C IndexedDB API, it became pretty clear to me that an agent could just grind through the test suite, and if it got close to 100% then I could be reasonably certain to have a bug-free implementation. But the only reason this works is because the Web Platform Tests are a phenomenally good test suite, honed by years of independent browser implementers discovering odd bugs and adding test cases for every unlikely scenario you can think of. Most companies, in their first-party codebases, could only dream of...