LLM Benchmark Design: Building the Ed-o-meter — Reinvently

← All posts

At a glance

Featherbench is a compact, single-file harness for testing LLMs directly,comparing model quality, latency, cost and refusal behaviour.

Provider and model variables are controlled, versioned so the comparison is reproducible.

Refusals from models are counted as failures

Its 28 tasks test simulate realworld usage scenarios rather than focusing on a single area.

Task variety is prioritised over repeated trials: five categories weighted toward realworld use, each run at least once, with the resulting uncertainty shown honestly rather than smoothed away.

Every model launch arrives with a wall of benchmark scores: MMLU, GPQA, SWE-bench, an alphabet of acronyms that grows with each release. The numbers are always fascinating. The problem is that each score measures expected performance in domains other than yours. A model can top a maths reasoning suite and still fumble the tasks your team actually runs.

The question that matters in production is more specific, because LLMs get expensive at scale. Once something runs millions of times a month, the pressure is to minimise tokens, spend and latency by running the smallest model that still clears your bar, that is "good enough" for your application rather than "good enough" to top a public leaderboard or be AGI. We made that argument at length in our original GLM-5.2 vs GPT-5.5 comparison; this post is about what happened when we acted on it.

The short answer: Featherbench is the harness — a single Python file that runs 28 fixed tasks against any model and grades them. The Ed-o-meter LLM Leaderboard is what the harness produces. One variable changes between runs (the LLM), every model judges every response blind, and the raw results are made public for maximum transparency.

There is a second reason to run your own. Increasingly the public suites are a question the models have already seen: benchmark maxxing is a documented problem, and labs under real competitive pressure have every incentive to optimise for the suites everyone reports. That is Goodhart's law applied to evaluation: a benchmark everyone trains against stops measuring what it was built to measure. Simon Willison's pelican-riding-a-bicycle SVG test is probably the best-known public illustration: it worked precisely because the scene was too absurd for any model to have memorised a good answer, and it's now famous enough that there's a live debate over whether labs have started training against it specifically. A private task set, written after the models shipped and embedding documents that exist nowhere else, is immune to this.

So I ran the measurements myself, on my own under-the-radar realworld tasks. Everyone already vibe-checks a new model, pasting in a few favourite prompts and forming an impression. This isn't more scientific than that so much as it adds a modicum of rigour: the same prompts every time, a written-down pass bar, costs and latencies recorded, and the whole thing repeatable when the next model drops.

What Does It Mean to Hold One Variable Constant?

Nothing changes between runs other than the model under test. The 28 tasks, the prompts, the answer keys, the checkers, the rubric, the scoring and the routing are all fixed and tracked in source control, so any two runs differ by exactly one thing. That is the whole design constraint: if a score moves, the model moved it, not a quietly re-tuned prompt or a changed grader. It sounds obvious, but most informal model comparisons vary the prompt and the model at the same time and then attribute the difference to the model.

Why Build a New Eval Harness?

There are excellent eval frameworks already. Inspect AI, promptfoo and Braintrust are all more capable than what I built, and if you need tracing, dataset versioning or a UI, use one of them.

My requirement was narrower: I wanted to be able to run my own realworld tasks, like get a sour dough recipe for my mum, audit the entire measurement path. When the results look odd, I can review the code myself, no need to log a ticket with a support desk to tell you what's happening in their proprietary solution. So Featherbench is a single Python file, just over 1k lines, with a dependency on jinja2 so I can format a report at the end. Tasks are JSON, not code, so they diff cleanly in review and a non-engineer can author one. Three provider families run through the same scaffold. It's MIT-licensed, since a result that can't be reproduced is a claim, not a measurement.

The models are called through OpenRouter rather than each vendor's API directly. Lazy or clever, I wanted to offer a way to eval models using one API key and one billing relationship instead of registering with a dozen providers. This makes adding a model to the panel a one-line change rather than a new integration project. But it earns its place on measurement grounds too. Open Router offers pretty much every model, whether a frontier...