AI Software Development – What Does The Data Say? – Codemanship's Blog

Skip to content

I’m currently pulling together a bunch of sources – that are mostly recent – on the topic of LLMs and their use in software development.

Some are peer-reviewed studies. Some are industry studies that haven’t been peer-reviewed.

One is statistical physics. Expect more from that angle. Wanna’ know the limits of a technology? Ask a physicist.

One is just a blog post, but very useful information about the effect of context size.

Most are corroborated by personal experiments and also observations on teams. As time goes on and more data comes in, my picture comes more into focus.

Before I cite the sources, a quick executive summary for all you busy executives out there:

Truly autonomous and reliable long-horizon agentic software development is so highly improbable using LLMs that it’s essentially science fiction.

The maximum effective context limits of LLMs – including hyperscale "frontier" LLMs – beyond which model outputs become unusably inaccurate is orders of magnitude smaller than advertised limits. The most common mechanism for extending inference over large contexts is what vendors call "compression". This means that parts of the context are summarised by the model, which is a famously unreliable/lossy process.

LLMs cannot distinguish between recent and out-of-date information in the context, and information in the model itself, learned during training ("dominant priors"), can often "outweigh" information we give it. To an LLM, it’s all just tokens, weights and probabilities. Right, wrong, new, old – the highest probability wins. Big contexts and "attention dilution" – where probabilities in the context become too small to compete with the ones in the model – are likely to make these effects worse.

Repo-level .md files tend to make model performance worse, probably because they add noise instead of signal in many specific tasks. Model-generated .md files are especially problematic in this respect, it seems. Upshot: including your team’s coding standards and an architecture summary for every task is probably counterproductive.

LLMs struggle with negation. Telling them not to do something can often have the same effect as telling them to do it. In case you were wondering why some of your guardrails are about as reliable as a coin-toss.

LLM inference is more accurate when we give them examples (demonstrations) rather than just describing what we want. They’re pattern-matchers. Show them the patterns – more "like this" and less "do this" (and no "don’t do this").

Large/long-scale industry studies show a clear trend – output is up (more code, more commits, bigger diffs), but outcomes don’t reflect that trend. If anything, the average team is taking longer to ship worse software. If ever we needed proof that software development isn’t a production process… Some studies find a small % of teams getting modest gains in outcomes, and correlate that with their existing software development capability. AI coding is an amplifier of, not a fix for, development strengths and weaknesses. (You’d think organisations would be lining up to do something about that… Sigh.)

The psychological and cognitive factors in LLM use are a growing field of serious research. One study found a significant correlation between confidence in AI output and belief in the paranormal. Multiple studies found a negative impact on learning, cognition and critical thinking with greater LLM reliance. New research suggests that reports of developers feeling demotivated and burned-out with extensive use may have some real truth behind them.

Deep neural networks, including LLMs, struggle to learn patterns with long-range dependencies, at any scale of model. They will always be "driving in fog", with local, short-range probabilities crowding out long-range ones. In case you were wondering why they suck at the "big picture" – probabilistically, it’s a blur.

The energy and compute needed to train an LLM to be an order of magnitude more reliable – e.g., wrong 3% of the time instead of 30% – is 10^20 times what the current frontier models require. Don’t expect significantly more reliable models any time soon. Any future gains in reliability will have to made by better context engineering (deciding what to include in the input) and more effective quality gates deciding what to do with the output- and that’s exactly what we’re seeing AI companies focusing on these days. Models may get more powerful, but not significantly more reliable. This it folks – work with what you’ve got!

Some AI champions will protest research that points to no significant improvements in model performance by pointing to the many published benchmarks that do indeed show LLMs getting better and better. But other research finds that we might wish to be more skeptical of benchmark performance, partly because many of the most popular ones measure what’s easy to measure algorithmically –...