LLM Performance and Following Requirements – Dan PalmerThere&rsquo;s a disconnect between the experiences different people are having with LLMs. Some find them transformational, others find them incapable. Partly this is down to skill, partly this is down to a dislike of the technology and it&rsquo;s social implications, but I think there&rsquo;s a further issue. This post is not trying to be pro-AI or anti-AI, I think the social issues are real, I don&rsquo;t believe LLMs live up to all of the hype, but I do find using them can be extremely productive.<br>So what&rsquo;s the problem? It stems from how LLMs meet requirements, and possibly how humans express them. Let&rsquo;s do a review of where LLMs work well and fall short to observe some patterns&mldr;<br>Where LLMs work well<br>This is not to say that LLMs work perfectly for all of these things, but rather that they generally work better, and need less guidance.<br>Vibe coding – by this I mean deferring all programming to an LLM, not writing any code, not reviewing any code, and focusing on one-shot or few-shot implementations. Clearly LLMs are very good at this, and an ecosystem of products has emerged around it.<br>Security testing – LLMs are also seemingly amazing at finding vulnerabilities in code and systems. The longer the trajectories, the more capable LLMs are at finding vulnerabilities.<br>Short creative works – LLMs excel at things like poetry and songwriting. There&rsquo;s a separate question of artistic value1, but it&rsquo;s undeniable that LLMs can produce perfectly reasonable poems, songs, short form creative writing, and change the voice or nature of existing documents.<br>Where LLMs fall short<br>Similarly to where LLMs work well, this is not to say that LLMs fail on these next areas, but rather that they struggle, perform worse (in quality, time, tokens), or require more guidance and human vigilance to achieve the desired outcome.<br>Code maintenance – LLMs are not good at keeping codebases in good condition, being generally poor at balancing forward progress with maintenance.<br>Writing for a technical audience – LLMs produce poor technical writing, almost always failing to convey the correct level abstraction, missing nuances, or misunderstanding the amount of historical context that should be included.<br>Evolving a codebase – Producing code over time is very different to producing it once, needing to account for cross-system dependencies, backward compatibility, data migrations, deployment cadence, and even building confidence in changes. LLMs struggle with all of these.<br>Research – LLMs often appear to be great at research, whether it&rsquo;s researching how a codebase works, or researching a topic and preparing a report, but as any expert can attest to, they often miss subtleties and produce text that may be technically valid, but emphasises the wrong things or tells a story in the wrong way such as to miss the point.<br>Drawing out a pattern<br>My hypothesis is that the pattern between the good and the bad is down to requirements.<br>LLMs work well when there are few requirements, and struggle when there are many. They also work well when the requirements are clear, but I don&rsquo;t think that&rsquo;s the main driver.<br>Let&rsquo;s take vibe coding as an example. There are very few requirements given with vibe coding. The user specifies some high level goal, and the model is free to do whatever it wants to get there. Does a React frontend work well, or match the training data more? Then let&rsquo;s use a React frontend. Is a Postgres backend close to the median in the training data, let&rsquo;s use that. The user doesn&rsquo;t care, and the model is free to run wild, introducing new dependencies anywhere, multiple ways of doing the same thing, and all manner of other engineering sins.<br>What about security? Finding security vulnerabilities is hard, but we actually lack much data about the counterfactuals. How many runs with LLMs failed to find anything? This field works well because it is fundamentally unconstrained, and again the model is free to explore. The actual exploit also doesn&rsquo;t need to be well built, it only needs to work.<br>Creative works similarly tend to have few requirements, all explicit in the user prompt. The space of valid outputs is large.<br>On the other hand, LLMs struggle with evolving codebases over time. A codebase is essentially a collection of requirements – which APIs are available and should be used, how each subsystem works, schemas, and so on. If one were to exhaustively list the constraints on a codebase, it would be a vast trove of information, and would likely blow up the context window far beyond what is practical.<br>Many of those requirements are also very implicit. Backwards compatibility is a subtle issue and can be hard to measure. Performance requirements are implicit and hard to measure. And a codebase is not fixed in time, but development requires a complex series of commits, deploys, and migrations to evolve it over time.<br>Writing for...