The Glass Is Half... Correct? Half Our SLM Benchmark 'Failures' Contained The Right Answer

neurometric’s Substack

SubscribeSign in

The Glass Is Half... Correct? Half Our SLM Benchmark 'Failures' Contained The Right Answer<br>An analysis of small model failures on CRM arena

Rob May<br>Aug 05, 2026

Share

What we ran

Three models, two tool interfaces, 340 CRM tasks each — 2,040 agent rollouts in total.<br>The benchmark is CRMArena, run through the Harbor/dockworker harness. Every task asks a question about a read-only Salesforce org (”in May 2021, which state had the quickest case closures?”, “which knowledge article does this quote violate?”) and grades the submitted answer by exact match. Seventeen task categories, split evenly across a B2B and a B2C org.<br>The interesting design choice is that the same 340 questions are posed twice, behind two different tool surfaces:

Identical questions, identical ground truth — we verified this holds on every task where both suites produced a graded answer. So the two suites isolate one variable: does the model do better writing SQL, or picking the right pre-built tool and filling in its arguments?<br>The three models: gemini-3.6-flash and gemini-3.1-flash-lite (hosted), and gemma-4-E4B-it (open weights, served locally on vLLM). One is a small local model; two are hosted. Every trial used the same noshell agent scaffold, whose only way to answer is to call a terminal submit_answer tool.<br>The scoreboard

Read it and you’d conclude the ordering is obvious and the small model isn’t close: 39.7% against 67.6% is a 28-point gap. Every pairwise difference here is statistically significant (p ≤ 0.009).<br>Then you look at how the failures happen, and the picture inverts.<br>Half of the small model’s “failures” contain the right answer

Not every zero is a wrong answer. A trial scores zero if the answer file was never written at all — and the scaffold only writes it when the model calls submit_answer. Classifying all 2,040 trials by how they failed:

More than half of gemma’s SQL trials never submitted anything. And they didn’t crash or time out — in 171 of them the model finished its work, wrote the answer out in prose, and simply never called the tool. Something like:<br>The case_metrics call returned an average closure time of 4.2 days for CA, the lowest of any state. The answer is CA.

Graded: zero.<br>So we tested the obvious question — were those prose answers right? Ground truth isn’t recorded for ungraded trials, but because the same 340 tasks appear in both suites, we could recover the expected answer for every one of them and grep the final message for it.<br>Of gemma’s 250 prose non-submissions, 44–52% contained the correct answer . The range is the strict and loose reading of the same check: the loose count is any trial whose prose contains the expected value; the strict count additionally requires that the model wasn’t hedging across a list of candidates (≤1 other record ID mentioned). Both bound the same conclusion.<br>That reframes the scoreboard as a lower bound. Fixing one scaffold behaviour — get the model to call the tool — moves gemma to:

A model that looked like it scored 26% on SQL was doing work worth about 50%. Its measured number was roughly half its actual competence, and every point of that gap is a formatting bug.<br>The same correction barely moves the hosted models — gemini-3.6-flash has exactly zero prose non-submissions, and gemini-3.1-flash-lite has one. They always call the tool. What we were measuring, for a third of the benchmark, was instruction-following on the harness contract, not CRM reasoning.<br>There’s a cleaner way to see it. Restrict to trials that submitted anything, and the reasoning quality behind the scoreboard separates from the plumbing:

On answers it actually submits, the small local model is within two points of the hosted flash-lite model — a difference well inside the noise at this sample size. The 11-point headline gap between them is almost entirely tool-calling discipline.<br>What it costs

This is where the small model stops being a curiosity. Token totals are the whole run; the per-win column divides by correct answers, which is the number that matters if you’re paying for throughput.

Gemma on the tool API buys a correct answer for 45.4k tokens. gemini-3.6-flash needs 196k for the same thing — 4.3× more . On SQL it needs 598k, or 13× gemma’s best configuration .<br>The driver is visible in the reasoning-token column. gemini-3.6-flash spent 1.36M reasoning tokens on the API suite and 2.59M on SQL. The other two models spent none. That’s what the extra accuracy is bought with: 3.6-flash’s median trial emits 3,978 output tokens on the API suite against gemma’s 501 — an 8× difference in generated text per attempt.<br>And time to completion — median wall time of the agent execution phase alone, excluding container build and verification:

Here the small model does not win, and it’s worth being precise about why. gemma is 12.6s per attempt against 3.6-flash’s...