Use the right model for every task | Gerard Rodes
Use the right model for every task
Choosing a good enough model for programming
2026-08-07
(you can't)<br>Running models is expensive and we have this dream of being able to "use the right model for every task", so we can be cost-effective.
The problem is: How do you determine which is the right model for a specific task? What does it even mean to be "right"? Does it mean fast, cheap, smart, something else? A mix of everything?
How do you know that a more capable model wouldn't have solved the problem in fewer turns (and ending up costing less), or that it wouldn't have avoided that non-obvious mistake? How do you know that a less capable model wouldn't have produced the same output faster and cheaper?
You can't know, because the only way of answering that question is to run both models and compare them yourself. By that time, you've already paid for both executions anyway.
There are routers that promise they can categorize the complexity of a prompt and route it to the cheaper model that can solve it. The problem is that categorizing the apparent complexity of a prompt is not the same as knowing which model will actually solve it correctly. A cheap router can make a prediction, but it cannot know whether that prediction was right without evaluating the resulting answer against something more reliable. At the same time, a real complexity evaluation of a problem might be as expensive as solving the problem itself.
These routers will be upfront and tell you something like "we have a 50% reduction in cost with a 5% quality loss". That might be true for their specific benchmark workload, which they have been tuning their router against. But they cannot guarantee that you will not get a much higher loss without running your entire workload through their most capable model and calculating your specific loss.
In the end, I have found a single valid approach: Always use the model with the highest expected quality you are willing to pay for. Anything else is just betting your time (and money) on a cheaper model being good enough, without knowing whether it actually is.