Gemma 4b vs Gemini Flash: You Don't Need Frontier Models For Tool Calling Workflows

neurometric’s Substack

SubscribeSign in

Gemma 4b vs Gemini Flash: You Don't Need Frontier Models For Tool Calling Workflows<br>What our Acebench analysis showed about the models

Rob May<br>Aug 11, 2026

Share

Most of what makes an AI assistant useful isn’t prose — it’s calling tools: booking the meeting, adding the cart item, pulling the shipping estimate. When these products fail, it’s rarely bad writing. It’s the wrong function, a mangled date, or three calls when one would do.<br>AceBench measures exactly that: ~2,000 hand-annotated tasks drawn from ~4,500 synthetic APIs across eight domains (finance, health, travel, tech, entertainment, and more), published January 2025 and later accepted at EMNLP.<br>Three things make it worth a practitioner’s attention:<br>Grading is mechanical and brutal. Right function, right call count, right arguments — or it’s wrong. No partial credit, no AI judge. One bad field voids the form. Harsher than real life, but reproducible.<br>It comes in tiers. Normal (clear requests), Special (vague), Agent (multi-turn). We ran Normal, English only — this says nothing about ambiguity or agent loops.<br>It’s designed to be taken apart. Tasks isolate specific failure modes: value types, near-duplicate functions, mid-conversation drops. A leaderboard number says which model wins overall. A benchmark that comes apart says whether a cheap model is good enough for your work — usually the real question.<br>What we ran

Five models, 772 tasks each, 3,860 attempts, English Normal split:<br>gemini-3.6-flash, gemini-3.1-flash-lite — Google’s hosted models

gemma-4-12B-it, gemma-4-E4B-it, gemma-4-E2B-it — open-weight, self-served (big/small/very small)

Same harness and tasks for everyone. Tool calls return a bare acknowledgement — no feedback, one shot per call.<br>Key point: the open model on hardware we control tied Google’s hosted model, using about an eighth of the words.<br>The scoreboard

The top two are a real tie: 45 tasks the hosted model got and the open one missed, 35 the other way. That’s the signature of two equally capable models, not a better and a worse one. Run it again and the order could flip. Every other gap in the table is real — but the tie at the top is between a hosted frontier model and a 12B open model on a single GPU.<br>What it costs

Scoring the same isn’t interesting. Scoring the same this cheaply is.<br>ModelWords per correct answer*:

*tokens (~¾ word each), total output<br>The hosted model generates ~8x more text per correct answer than the 12B model it’s tied with — almost all invisible “thinking” (655,000 tokens across the run). The other four models did none of that: read, call, stop.<br>And the thinking isn’t buying anything. When the hosted model got a task wrong, it generated more than twice as much text as when it got one right. Extra effort here is a sign of being stuck, not a way out. On a benchmark where the winning move is two steps, there’s not much to think about.<br>Generated tokens are the expensive half of any pricing page, and what determines latency. So: same accuracy, a fraction of the tokens, and a deployment you own instead of rent.<br>The catch

Cheap on tokens ≠ cheap in practice:

The 12B model that ties Google’s best was 4x slower per answer than the API it ties, despite generating far less text. The smaller E4B, on the same GPU, hit 5.6s at ~91% of the top score. Note: these numbers are default vLLM, full BF16, on a single L40S — plenty of room to push both further with better hardware.<br>The smallest model is better than its score

gemma-4-E2B-it came last at 67%. Look at how it failed and half the gap disappears.<br>Its top mistake wasn’t picking the wrong tool — it was making the right call plus extra, already-completed ones from earlier in the conversation:<br>Called: ask about gift etiquette → send the gift → schedule the meeting Wanted: schedule the meeting

It knew the current turn. It just also re-did two things already done. Graded: zero.<br>72 of its 255 failures end with exactly the right call, buried under repeated history. Score only the current turn and it jumps from 67% to 76% — close to Google’s smaller model. That’s a prompt fix, not a bigger model — and it barely moves the other four (0-3 failures each of this kind).<br>Where bigger models still earn their keep

Three places:<br>Nested arguments. A tool wanting {"journey": {"from": "Shanghai", "to": "Hangzhou", "times": {...}}} trips up everyone — best models barely clear 60%, versus 85-96% for flatter arguments.<br>Several calls at once. Same tool, three times, different details: the hosted model pulls ahead ~5 points. Coordinating calls is harder than making one.<br>Long conversations — the sharpest split. Four of five models improve turn over turn as context narrows the options. The smallest model collapses:

That last column is what matters in production: a conversation only works if every turn lands. 87% per-turn becomes 74% overall.<br>Two models...