Public LLM benchmarks are mostly garbage | GrandpaCAD
Back To Blog
Other Blogs<br>UpdateJul 5, 2026<br>1 min read
Organic Mode just got sharper, faster, and more reliable
FeaturedMay 19, 2026<br>3 min read
The cheatsheet I made so my grandpa can use AI CAD
Apr 18, 2026<br>7 min read
How to make money with a 3D printer (the boring math)
UpdateMar 29, 2026<br>4 min read
Stop Resizing AI Models. Set Your Printer Config Once.
UpdateMar 4, 2026<br>3 min read
We upgraded to Gemini 3.1 (and why we changed our mind)
FeaturedFeb 21, 2026<br>3 min read
Introducing Organic Mode
Feb 20, 2026<br>3 min read
Why we're not upgrading to Gemini 3.1
UpdateFeb 17, 2026<br>3 min read
Free 3D printing tools and calculators
UpdateFeb 13, 2026<br>4 min read
Blender vs OpenSCAD vs JSCad vs JSON: Choosing the best LLM-to-CAD engine
UpdateJan 29, 2026<br>2 min read
Streamlined 3D Printing with BambuLab
UpdateDec 12, 2025<br>2 min read
Image to 3D Model: Sketch It, Scan It, Print It
FeaturedNov 25, 2025<br>3 min read
Multi Color 3D Printing with AI
Nov 24, 2025<br>3 min read
How It Works
Nov 22, 2025<br>5 min read
Comparing State of the Art LLMs for 3D Generation
TutorialNov 21, 2025<br>2 min read
How to Make Your Models Adjustable
Nov 5, 2025<br>8 min read
How we test the 3D modelling agent
UpdateOct 19, 2025<br>4 min read
Migrating to Blender
Sep 23, 2025<br>4 min read
Welcome to GrandpaCAD Beta! Here's What You Need to Know
FeaturedAug 16, 2025<br>Living doc
Changelog
Jul 30, 2025<br>5 min read
Paddle vs. Polar for AI Subscriptions and Usage Based Billing
Jun 30, 2025<br>4 min read
GrandpaCAD: My Grandpa, AI, and the State of CAD
Jun 14, 2025<br>4 min read
Giving Grandpa His Creativity Back: How AI Helped Him Create Again
Jun 1, 2025
GrandpaCAD: For Press and Media
Public LLM benchmarks are mostly garbage
A friend asked me last week how Opus 4.7 was holding up on GrandpaCAD. I dismissed him. Here's the data I dismissed him with:
OpenRouter throughput. Gemini 3.1 sits around 62 tokens/second p50. Opus (when I last checked) was around 45.
Design Arena 3D Benchmark. Kimi K2.6 has 1369 ELO, Gemini 3.1 has 1320, Opus 4.5 has 1299.
Token prices. Opus tokens cost roughly 3x more than Gemini, 10x more than Kimi.
Three independent sources, all pointing the same way: faster, better, cheaper somewhere else. To be fair, I'd been dismissing Claude updates for a while. Sonnet 4.5 lost to Gemini on this exact workload last year, and I never even bothered running an Opus model because the per-token price scared me off before the eval did. Claude Code also feels sluggish in my day-to-day work, which only reinforced the story. So why bother running the eval?
I ran it anyway. The numbers came back upside down.
What the eval suite actually showed
I ran my standard eval harness against four frontier models: Opus 4.7 on auto thinking, Gemini 3.1 on medium thinking budget, GPT 5.5 with service_tier: priority, and Kimi K2.6 on Baseten (the fastest provider I could find for it).
MetricOpus 4.7Gemini 3.1GPT 5.5Kimi K2.6Weighted score0.5870.5560.5010.545Adherence0.5840.6140.5910.481Pass rate85.7%76.2%90.5%66.7%Error rate9.5%0.0%0.0%14.3%Code retries (avg)0.190.240.100.52Avg duration0m 32s1m 32s1m 46s0m 53sAvg cost$0.10$0.21$0.94$0.02Total benchmark cost$2.04$4.48$19.79$0.51
A few things stand out.
Opus 4.7 is the fastest. 32 seconds per generation. Gemini 3.1 takes 1m 32s. GPT 5.5 takes 1m 46s. The OpenRouter throughput chart says Gemini is faster on tokens-per-second, and that's true if you measure tokens. But thinking models burn tokens you don't see, and what actually matters is wall-clock time on your prompt. Opus thinks less, ships sooner. I can run three Opus generations in the time Gemini finishes one.
Opus 4.7 has the highest weighted score. 0.587, ahead of Gemini 3.1 (0.556) and GPT 5.5 (0.501).
Opus 4.7 is half the cost of Gemini, one tenth the cost of GPT 5.5. $0.10 per generation versus $0.21 versus $0.94. Token-price comparisons don't account for thinking budgets or how many tokens each model actually spends. Per finished 3D model, Opus is the cheap one.
Why 3D is the highest-signal LLM benchmark you can run
This is the part I keep coming back to. If you read text from GPT 5.5, Opus 4.7, and Gemini 3.1 side by side, you genuinely cannot tell which one is smarter. They all sound competent, they all hold the thread, and the differences hide in places you won't notice for months: subtle factual drift, quiet bias, reasoning that looks right and doesn't quite hold under load.
Code is a step better. Code that doesn't compile, doesn't compile. You catch logic errors at the unit-test boundary. But there are still bugs that hide for months because they only fire on specific inputs.
3D is different. The hook either connects to the wall plate or it doesn't. The phone stand either holds the phone or it tips over. The chair legs either touch the ground or they float. Your eyes catch broken 3D in 200 milliseconds, the same way you spot a typo in your own name....