My local LLM scored 6/6. It was wrong every time

My Local LLM Scored 6/6. It Was Wrong Every Time. · Mark HallSkip to content

One of the tasks in my benchmark asks this: How many 5-person committees can be chosen from 12 people? Reply with only the integer.

My 1.2B local model answered 60. The correct answer is 792. My evaluator passed it. It passed all six questions in that probe. The model had gotten every single one wrong. The scorer checked whether the answer looked like a bare integer, never whether it was the right one — so 6/6 passed, 0/6 correct . I had not made a small model smarter. I had built an instrument that could be flattered by good formatting. What I actually found Here’s where I landed after six months, before I walk back through how I got here. QuestionAnswerDid scaffolding raise closed-book knowledge scores?Not in my tests. Frozen-runtime MMLU-Pro: raw 10/20, same-prompt direct 10/20, harness 10/20. An earlier 13/20 did not reproduce.Did it raise verified, tool-backed task completion?Yes, substantially. On a sealed 100-case bank, Gemma 4 26B went from 5 to 56 completed cases with the same fixed model.Does more completion mean more reliability?Not automatically. The 1.2B model went 12 → 32 cases and stayed at 1 of 25 complete task families.Did the underlying model matter?Yes. Complete families: 1.2B held at 1/25, an 8B went 3 → 9, Gemma went 0 → 10.What was the mistake underneath all of it?Measuring answer shape instead of answer value, and treating a knowledge benchmark as a product benchmark.Everything below is how I got each of those wrong before I got them right. Starting with the easy wins I started with a hypothesis: a well-designed harness on a small local model could handle most ordinary tasks and feel like something much larger. Not because the model got smarter — because the system around it did. The early evidence was encouraging, and in retrospect that was the problem. Ask a small model whether to walk or drive to a car wash 50 meters away and it fumbles. Structure the reasoning — what is the user trying to accomplish, what does that require, what follows — and it gets there. Ask how many R’s are in “strawberry” and it needs a counting tool, not better spelling. Ask what’s heavier, a pound of feathers or a pound of gold, and watch it convert gold to troy ounces, convert again, and solemnly compare two numbers that were never different. Hand it a unit converter and the trap closes. Three puzzles, three fixes, and a growing sense that this was going to work. Then I asked an AI to make the tests pass I wrote about fifteen logic tests and a rough scoring rubric, and told Codex to iterate until everything came back correct. It did. That was the problem. Opus and Codex both have the same habit: they find the easiest path to the correct output. In this case, the easiest path was hard-coding the answers because I’d handed them the correct information. Imagine my dismay when I caught them red-handed in code review. A stern talking-to only goes so far. I wagged my finger and told them to knock it off. They obeyed — begrudgingly — and after many, many attempts, I got to something like the right shape. But it was brittle. Too much of it leaned on regex formatting; word a question slightly differently and everything would break. It made me want to pull my hair out. I was, without knowing the term for it yet, overfitting — with an enthusiastic assistant doing the overfitting on my behalf. The realization that reframed everything I was reading benchmark results for a newly released model when the thought landed: I could not improve anything until I could measure it. Obvious in hindsight. Embarrassing in the moment. I had spent weeks generating ideas and evaluating them against a rubric I’d written casually, with an agent that would happily satisfy it by any means available. Every conclusion I’d drawn was downstream of an instrument I had never validated. So I went looking for a real benchmark and landed on MMLU-Pro, a more difficult version of the MMLU benchmark everybody quotes. Closed-book, widely used, publicly reported. I built a workbench around it and set the agent loose. MMLU-Pro was the wrong shape and I didn’t know it The early results looked good, because early results always do. Then they flattened. I ran it for days. Movement was small, inconsistent, and never reproduced cleanly. Eventually the controls told me why. Under a frozen runtime: ConditionResultRaw baseline10/20 — 50%Same-prompt direct10/20 — 50%Current harness10/20 — 50%Historical product version10/20 — 50%Identical, every arm. The earlier 13/20 did not reproduce once the runtime was pinned, so it’s a historical observation, not an uplift. A separate experiment with sampled voting made things actively worse — 37.9% → 27.9% — and that mechanism was removed. I’ll be straight: I didn’t understand what MMLU-Pro was for. It is deliberately closed-book. It measures what a model can produce from its...

My local LLM scored 6/6. It was wrong every time

Related Articles

(no title)

Scientists reverse brain aging, with a nasal spray

AI has torched the market for junior programmers

Al Vigier: Canada's AI strategy shouldn't include secret Palantir bills

The Anatomy of an AI-Native Org