Reviewing AI Code Is Not A Viable Argument | Musings about software
Reviewing AI Code Is Not A Viable Argument
18 Jul 2026 - Thomas Depierre
I am a skeptic of the utility of LLM in software development. It is not because<br>of IP laws problems (even if they are highly problematic), nor is it for<br>ecological and ressource comsumptions reasons. It is not even for “they are all<br>crap” reasons. My problem with LLM Coding Assistants is that I cannot see, in<br>the face of the scientific evidence, how they can help someone write code better<br>or faster.
And the thing that irritate me is that it seems that none of the proponents of<br>LLM Coding Assistants seems to ever address this problem and this evidence when<br>they defend their tooling choices. Worse, it seems that they give fuel to my<br>arguments everytime they write a rebuke of the skeptics. So let’s look at what I<br>have a problem with, how empirical scientific research support that view, how<br>proponents of LLM Coding Assistants could show this is not a problem and then<br>how right now they are doing the actual opposite.
Note: This piece was written nearly a year ago, hence why you may find<br>vocabulary, like “Coding Assistants” that has been mostly replaced at this<br>point. Sadly, I have not been able to find a term in the current vocabulary<br>around genAI used to code that covers all the use case. So I kept the “Coding<br>Assistants” wording.
The Intern Problem
The fundamental problem of LLM Coding Assistants that my criticism center on is<br>their relatively high risk of getting things wrong. For all kind of reasons,<br>some structural to how LLM works and other more akin to the interfaces we<br>provide to interact with them, LLM Coding Assistants get things wrong. It can be<br>hallucinations, typos, simply doing something that is not linked to the task<br>demanded, going into a different path, etc etc.
A lot of people I have talked about that experimented with LLM Coding Assistants<br>explains that they feel “Like an intern”. Like an intern, you should not expect<br>too much from them, you should expect that everything they do will be more or<br>less wrong, and that they have no idea what they are doing, but are highly<br>enthusiastic. I see they never got me as an intern. I was definitely not<br>enthusiastic.
And their answer to that problem, the one you will see all over the internet, is<br>simple. You just do the same thing that you do with interns and junior<br>developers in your team. No, they do not mean you put everything they did in the<br>bin and forget about it. What they mean is that you should review all the code<br>yourself. I mean, you are the human that know better. And you are the one<br>responsible for the code anyway. And on top of this, you are doing that for all<br>the code that get into your codebase anyway, you do not let code get in without<br>a review, right?
What We Means When We Say Reviews
First of all, I want to be clear here. There are different practices in the<br>litterature and in the profession grouped under the term “review”. So let’s be<br>explicit here. Seeing the degree of (mis)trust and potential mistakes there, we<br>should not accept the kind of “lightweight and heavily distributed” reviews that<br>we see the most in our industry as the standard for LLM Coding Assistants<br>supervision by a professional developer. They are not a bad thing to do, nor are<br>they inefficient, but they have been shown in the litterature mostly as good to<br>distribute knowledge of changes and as a way to enforce all kind of surface<br>level rules.
For AI Coding Asssistants, we will need a proper “code review”. Not something as<br>formal and complex as reviews of old, by committees, painstakingly checking<br>every line one by one over a few hours. But still, we want something quite<br>involved and complete. After all, these are interns writing sometimes highly<br>complex code. And if there is something we know in software, it is that the<br>devil can be in the details.
The Limits Of Reviews
Without going into some philosophical depth of reviewing as a practice, there is<br>a glaring problem in this idea. From all the research we have, we have learned,<br>empirically a few things about code reviews. And the evidence is relatively<br>solid here, within reasonable limits. You will see that these do not matter<br>there.
A review that last more than 1h is too long.
A review that has to be effective cannot be more than 400LOC at a time, in<br>that time.
Empirical research has shown that reviews that are longer than 1h quickly reach<br>diminishing returns whatever is the size of the code being reviewed. So this<br>is not only that people cannot find bugs anymore after 1h because they already<br>thoroughly reviewed most of the code. No, it is more linked to the fact that<br>after 1h at that level of attention, people start getting tired, bored and<br>simply need some time off.
Of note is the total absence of research as far as I could find, on the recovery<br>time needed between review sessions of 1h. So I canot tell you how frequently<br>someone could do 1h review sessions....