It's time to rip off the band-aid and stop performing code reviews

Sign in<br>Subscribe

For everyone who just got a little tense reading the headline: I get it. I was there too, not so long ago. At the beginning of this year I spent a month building what was at the time–and might still be–the best AI-aided code review tool in the world. (The screenshot above is from that.)<br>And now I don't do code review in the classic sense at all.<br>The World is Changing<br>In human-driven software development, code review served three primary purposes:<br>Knowledge dissemination. It's not good if only one engineer has the understanding required to make changes to a given piece of code without breaking things. Code review is a good way to ensure that at least one other member of your team understands the code, and the "whys" behind it.<br>Training. Having a senior engineer review a junior's code helps the junior improve for next time.<br>Defect detection and technical debt avoidance. Conventional wisdom is that having someone who didn't write the code review it is one of the least expensive and most effective ways to do QA.<br>Code review is no longer necessary for the first two, and only partially effective for the third:<br>We can get up to speed on what's going on in arbitrary code so fast with AI assistance that the tradeoff that code review makes (hours or days reading each PR before merging–"big review up front", as the agile people might say) no longer makes sense; just derive what you need when you need it in your favorite harness.<br>Models don't learn across sessions (although ~every big lab is trying to solve this and I expect they will in a single-digit number of years). And giving feedback to a junior engineer on code his AI wrote is just not valuable enough to justify the time.<br>Technical debt is nearly a nothingburger now. If you went in the wrong direction, just... change the code to go in the right direction. This is the YAGNI principle applied at a higher level: don't spend time doing expensive things now that might not even be necessary, when the cost of doing it later is no higher than doing it today. (Actually, with models getting smarter month to month, it will often be lower.)

In other words: the optimists were right. New models really can clean up technical debt created by last-gen models. Yes, they still make mistakes, but fewer with each new release. Fable 5, and to a lesser degree Sol 5.6 and K3, are very, very good at improving code in a way that we didn't see before.<br>That leaves defect detection.<br>How do you know if the code is any good, if you haven't read it?<br>First of all, let's level-set: code review was never amazing at finding defects. It was better than nothing, sure, but the best studies showed about a 35% defect detection rate, and often lower in specialized domains like security.<br>We're engineers. We automate things. How can we apply that to defects? (If only there were a vast empirical literature about software QA!)<br>You can write code to verify code<br>With intelligence-on-demand you have options that you didn't before. I'd go so far as to say that QA is one of the most exciting fields in software engineering right now. We have literally decades of research on how to make our code robust, even really hard-to-reason-about distributed systems code, and most of it was completely impractical in The Before Times because of the prohibitively high level of changes you needed to make to your project to structure or instrument it appropriately.<br>That problem just doesn't exist anymore. So it's crazy to me that almost nobody has started pulling in these techniques yet.<br>But code authoring is only part of it<br>I think many engineers have a bit of a blind spot here because the only time they use inference is in a coding harness or a web chat UI. But you can connect models up to all kinds of problem shapes!<br>For instance: besides distributed testing, another set of tools that has been around for decades and seen relatively low adoption is static analysis. Coverity has been around since 2002! But we gravitated instead towards simpler, less sophisticated tools like Ruff and Clippy because more ambitious checks were too noisy in real codebases to be useful.<br>But now you have a tool that can literally wade through the noise and decide what's actually relevant. (You might be tempted to just say, "Code is cheap so let's just fix all warnings everywhere," but this will still do ugly things to your code and make it worse, not better. Judgement is still required.) I've written two different custom analysis pipelines for Bifrost that both feed into GPT to classify which should be ignored, which should be autofixed, and which should be escalated to a human.<br>What then is the "right" way to spend your "review" budget?<br>Again, I'm not arguing that you shouldn't be involved with the details. Humans still need to understand architecture, invariants, and intent. What I'm arguing is that manually reading diffs, even condensed diffs, is no longer...