MediumI Built a LLM Benchmark Tool and Tested It on Free Models — Here’s What I Found | by Cheikh seck | Aug, 2026 | MediumSitemapOpen in appSign up<br>Sign in
Medium Logo
Get app<br>Write
Search
Sign up<br>Sign in
Golang
Llm Benchmarks
Command Line Interface
I Built a LLM Benchmark Tool and Tested It on Free Models — Here’s What I Found
A lightweight, dependency-free CLI that measures accuracy, instruction following, tool calling, and structured output across any OpenRouter model.
Cheikh seck
5 min read·<br>Just now
Listen
Share
The Benchmark Problem Nobody’s Solving<br>Every week, a new LLM drops with a headline score that makes previous models look obsolete. MMLU, HumanEval, GPQA — these benchmarks test PhD-level knowledge and competitive programming. They’re impressive. They’re also irrelevant to most of us.<br>The average user doesn’t need a model that can ace a quantum mechanics exam. They need one that follows instructions reliably, outputs valid JSON when asked, calls the right tool at the right time, and doesn’t hallucinate basic facts. Most benchmarks don’t measure any of that.<br>So I built a tool that does.<br>Press enter or click to view image in full size
The Benchmark Tool<br>I wrote a Go CLI that hits OpenRouter’s API across four suites:<br>┌─────────────────────────────────────────────────────┐<br>│ Suite │ What It Measures │<br>├────────────────┼────────────────────────────────────┤<br>│ Accuracy │ Knowledge + math (15 tasks) │<br>│ Instruction │ Deterministic follow-through (12) │<br>│ Tools │ Function calling (BFCL-style, 12) │<br>│ JSON │ Schema compliance (5 tasks) │<br>└────────────────┴────────────────────────────────────┘Each task runs 3 times for reliability scoring. The tool pulls live pricing from OpenRouter, calculates cost per correct answer, and outputs JSON, CSV, and Markdown reports.<br>Why OpenRouter? Two reasons. First, it aggregates dozens of models from different providers behind a single API — you can compare a Chinese model against an American model side by side without managing two SDKs. Second, BYOK (Bring Your Own Key) means I can use my own key, my own quota, and my own billing. No vendor lock-in.<br>And if OpenRouter is hiring — they should definitely hit me up. I’d love to work on the tooling that makes this kind of cross-provider benchmarking trivial.<br>The Models<br>I compared two free-tier models:
Press enter or click to view image in full size
The size difference is staggering — 550 billion parameters versus a “flash” model. Intuition says the bigger model wins. The data says otherwise.<br>The Results
Press enter or click to view image in full size
Accuracy (Knowledge + Math)<br>These 15 tasks test what a model actually knows — capitals, arithmetic, science, history — and whether it can produce the right answer. Unlike academic benchmarks that test PhD-level expertise, these are the kinds of questions a real user might ask on any given day. The results reveal a clear pattern: the smaller model handles factual recall with surprising reliability, while the larger model’s struggles with basic math and simple facts suggest that raw parameter count doesn’t always translate to everyday usefulness.
Press enter or click to view image in full size
Ling gets 11 of 15 right on at least one try. Nemotron gets 9. But the pattern is telling — Nemotron fails on basic math and factual recall consistently, while Ling’s misses are scattered.<br>Instruction Following
Press enter or click to view image in full size
Ling nails every single instruction-following task. Nemotron struggles with constraints — no commas, no digits, exact word counts. This is where the “general usability” gap is most visible. A model that can’t follow “don’t use commas” is a model that will frustrate real users.<br>Tool Calling
Press enter or click to view image in full size
Ling calls the right tools 10 of 11 times. Nemotron gets 8. The biggest gap: parallel tool calls (multiple tools in one request). Nemotron fails to call two tools simultaneously three times out of three.<br>JSON Output<br>Both models score 100% on structured output. This is the one area where they’re evenly matched.<br>How the Tool Works<br>The tool is a Go CLI that accepts model names, API tokens, and suite configurations via command-line flags, then sends each task to the OpenRouter API. It executes every task across multiple runs, scoring pass/fail for each trial while tracking latency, token usage, and cost — then aggregates the results into per-suite and overall scores. Finally, it writes three output formats (JSON, CSV, and Markdown) so you can inspect the raw data, import it into a spreadsheet, or share a readable summary.<br>┌─────────────┐ ┌──────────────────────┐ ┌─────────────────┐<br>│ CLI Flags │───▶ OpenRouter API ────▶ Live Pricing │<br>│ -token │ │ /v1/chat/completions│ │ /v1/models │<br>│ -models │ │ BYOK auth │ │ Cache │<br>│ -suites │ └──────────────────────┘ └─────────────────┘<br>│ -runs │ │<br>└─────────────┘ ▼<br>┌─────────────────┐<br>│ Task Execution │<br>│ 4 suites × N │<br>│ tasks × R runs...