The Bots Always Find Something · Avy Faingezicht

I have spent the last couple of months building a proof of concept for a new company. As a solo founder, I've been moving fast by leaning hard on AI. My army of agents both build features for me and act as another pair of eyes as the product evolves. Once I got past basic prototyping, I configured Cursor's Bugbot and Codex to automatically review every code change going into the product. This seemed prudent. After all, the software will manage other people's money. Increasingly, though, I think I am just wasting tokens by p-hacking my own codebase. The bots are built to find problems, so they find problems.

Run a reviewer over a changeset and it might flag something worth fixing. Run four reviewers with slightly different lenses, and an orchestrator that reviews their suggestions, and one of the five is bound to identify a problem. Likely, the bug will be a rare race condition inside an error path that only happens when an external API returns an undocumented response while another request is interrupted at exactly the wrong time. The explanation will probably be followed by a ridiculous line like: It's not an edge case, it's an invariant violation quietly hiding at the boundary between individually reasonable assumptions, which is exactly why it matters.

Technically possible. Practically irrelevant.

Back to the real world, the analogy to repeated statistical testing is obvious: If you run enough hypothesis tests, some will look significant by chance. This is why researchers worry about multiple comparisons. Although "false positive" is not quite the right term, AI code review has a similar problem. The findings aren't false, since the model correctly identified a bad thing that could happen, but the expected value of fixing that sequence is often near zero. No real user was ever gonna hit that bug.

Suppose a bug has:

a 0.01% chance of occurring,

affects 0.1% of users when it does,

and causes a mildly confusing UI state.

A reviewer may notice it. That doesn't mean we should spend an hour adding another state machine, test fixture, fallback, and comment to prevent it. In fact, the fix can easily make the system worse. More branches mean more code to understand, more states to test, more places for future bugs to hide, and perhaps most importantly, more tokens for every future maintenance run to burn.

What is the cost of leaving it alone? Is this code even gonna be around long enough that we need to worry about it? Does the proposed fix add more complexity than the bug warrants?

AI creates an inversion of the old PR review problem. Historically, code review required scarce human attention, so reviewers focused on things important enough to mention. We literally prepended "this is a nit but…" on things we didn't expect would happen often enough to matter. AI review makes diving into extreme detail seem free. The tempting answer is to 80/20 it: keep the findings that prevent most of the damage, ignore the rest. But the thing is we are not trying to catch 80% of bugs. We are only trying to catch the bugs whose expected cost exceeds the cost of the fix. A rare failure on a money-handling path can still be worth an hour or ten of your time to review or a bot to fix. A UI ghost in a prototype is not the same category.

The models have terrible incentives here. If I ask, "Find correctness issues in this code," a model that replies "looks fine" feels less useful than one that produces five detailed observations. There is no cost to the model for sending me down a 30-minute rabbit hole or burning an additional 1M tokens. Precision matters enormously to me and barely to the reviewer.

What used to turn junior engineers into senior engineers was the realization that not everything that can be fixed should be fixed. Let's put that in the next RLHF loop.

Thanks to Max Faingezicht, Lakshman Mody, and Nikhil Pai for their feedback on early drafts of this post.

Photo: Fire lane, by me. Previously posted in Time Travel Variety Pack.

Subscribe by email