Evaluating AI Agents as Products
Evaluating AI Agents as Products
Coding benchmarks evaluate AI agents on well specified tasks.<br>Open ended qualitative checks show which can make the nicest demo but<br>leave enough flexibility that they don’t measure the rigidity of real<br>work. Neither capture human-in-the-loop performance nor do they speak to<br>efficiency and productivity. To explore how we can better measure how<br>“useful” AI agents are, I introduce three tasks to generate data<br>labeling interfaces in an (ideally) short interactive coding session.<br>The specified interface functionality combines mandatory requirements<br>with open ended choices, allowing us to evaluate efficiency,<br>collaboration, and taste, providing a proxy for how well the agents<br>perform as tools in real coding situations. Here I introduce the tasks<br>and evaluate five agents – three smaller local models and two near<br>frontier.
Andrew Marble<br>marble.onl<br>andrew@willows.ai<br>Aug 15, 2026
When we started doing ML, benchmarks scores pretty closely captured<br>the performance of a model. If a classifier had a certain precision and<br>recall, as long as the data distribution didn’t change this told you<br>what you needed to know. I think AI’s origins in simpler ML models, and<br>the current narrow gap between academic advances and commercial<br>offerings has kept an outsized emphasis on benchmarks.
Benchmarks can be most useful when (a) we want to compare<br>models and (b) we are in a period of rapid model advancement.<br>And in the brief history of LLMs, model performance was probably a<br>suitable proxy for product performance. Anything you build on GPT-5.6 is<br>going to be better than if you’d built it in GPT-4. A known issue of<br>course if that benchmarks saturate and all models can basically “ace”<br>old benchmarks making them useless for comparison. Less discussed is<br>that the entire concept of model benchmarking may be saturating in many<br>applications, and the remaining alpha is in more product-like qualities<br>around user experience and the elusive “taste”.
The Artificial Intelligence Coding Agent Index1 is<br>a composite index of three public benchmark datasets (DeepSWE,<br>Terminal-Bench V2, SWE-Atlas-QnA). The first two of these have agents do<br>software engineering tasks with well defined expected behavior. The last<br>one has the agent answer questions about a codebase.
I would contend that such evaluations no longer align with how<br>effective a given tool (say Cursor or Codex or Claude Code) is at<br>improving developer outcomes (saving time, shipping more, etc). Well<br>specified tasks test for a different set of behaviors than more<br>open-ended real-world problems where taste and judgment are necessary2. Furthermore, we’re at a point where<br>most agents can basically solve any task when a human is in the loop.<br>Finding tasks they can’t solve becomes increasingly about tricking them<br>or identifying edge cases instead of testing which is more useful in the<br>real world. The relevant question is no longer “can it do this” like it<br>was in the early days, but how efficiently does it do it (with a human<br>in the loop), does it exercise good judgement that leads to a quality<br>product, does it provide helpful input as a co-worker, etc.
On the other hand, there is a danger in open-ended testing without<br>clear goals. Recently Andrej Karpathy suggested prompting for video<br>generation3, and there are analogous coding<br>focused tasks like one-shotting video games. These make cool demos but<br>are too open ended to realistically test utility. AI’s perennial problem<br>is that it’s great for demos but real world considerations take the wind<br>out of its sails. A Netlify blog post also compared how well models<br>could single-shot web design tasks4. This has a similar<br>problem, it doesn’t test against any subjective use criteria, and it<br>only measures single-shot performance, not interaction, so doesn’t<br>obviously predict how useful a tool will be when used with a human in<br>the loop.
To explore how useful different coding agents (consisting of a model<br>and harness) are, I wanted a set of tasks that was specific enough to<br>have constraints and necessary success criteria, while also involving<br>sufficient flexibility to let models demonstrate their taste. Crucially<br>I also wanted real useability criteria. I didn’t want to just build<br>something and say it passed the tests and looked well designed, I wanted<br>there to be the equivalent of a user that could decide if the end result<br>met their needs. The goal is not to fully simulate real use, but to find<br>proxy tasks that give a sense of how efficient and effective the tools<br>are in real situations by simulating meaningful interaction and success<br>criteria.
Based on these criteria I experimented with tasks requiring agents to<br>build software in support of dataset labeling or annotation. This is a<br>task I use AI for frequently in real life. It involves clear functional<br>criteria – a user is presented with a set of data points and needs to<br>make some determination(s) about each (add a label), so the...