Rules vs examples: how much does spelling things out help? | Cassis<br>Skip to content Blog<br>Docs

Talk to us

Back to stories<br>Rules vs examples: how much does spelling things out help?<br>Rules, metrics, or worked examples? We measured which context representations shorten an LLM's path to a correct answer, and what the wrong turns taught us about good context.

Aloÿs Augustin July 9, 2026

On this page

Summary

What we foundReproducibility

The setup<br>The first curve, and why it didn’t say much<br>Holding completeness constant<br>How to interpret this?<br>Is the gain from examples or just more context?<br>When more context gives worse results<br>Are we just testing retrieval?<br>What is each example adding exactly?<br>Where the noise lives<br>Accounting for the cost<br>The hidden costs<br>So which one do you choose?<br>It started from an intuition. Why are examples so often recommended to help LLMs perform well? For our brains, it’s clearly easier to adapt an example than to reconstruct a solution from scratch. Is it the same for LLMs? Does it help a model if it has less “thinking” to do to get from the information it’s given to the answer?

We call that remaining work the derivation distance : how much transformation and composition the model still has to do to get from the facts in its context to the answer. A general rule leaves a long distance. A worked example with the same shape as the question leaves almost none.

There’s a whole range of possibilities when editing context: one can state general rules, provide snippets for common patterns, provide full-fledged examples, and in practice it’s usually a bit of a mix. We set out to evaluate what works best. This study is focused on data analytics use cases. We expect the findings to transfer to other use cases, but that would need validation. It’s easy to run a similar experiment on your own use cases. I’d be interested in the results if you do!

This post is the story of how we approached that, because the wrong turns are also interesting. They surface what makes a good context.

What we found

Completeness comes first. If a business rule is missing, getting the right answer amounts to playing the lottery.

Shortening the derivation distance pays off on hard questions: adding a full example library raised Haiku from 70% to 82% and Sonnet from 81% to 93%.

Removing the rules and metrics beneath that library left standard accuracy unchanged and hard accuracy within one to four points, while cutting context by roughly 30%.

Opus was already at 98% with complete English descriptions. Metrics and examples added only one percentage point.

A few relevant examples were not enough. The gains appeared with a broad library covering many query patterns and the small conventions around them.

Consistency matters more than volume. A single contradiction can erase the benefit of additional context.

There is no universal best setup. Smaller models benefit from comprehensive examples. Stronger models perform well with leaner context.

Reproducibility

The code needed to reproduce all the experiments mentioned in this article is open source on GitHub.

The setup

We started from three public analytics datasets: jaffle-shop (a toy cafe chain), theLook (a synthetic e-commerce store), and Olist (a real Brazilian marketplace export). Each comes with a context layer that we initially built to test Cassis: table and column descriptions, business rules written in prose, and named metrics with their SQL. For each dataset, we also generated 12 standard questions and 11 hard questions with known answers.

The questions come in two tiers. The standard tier is straightforward: “what is our total revenue”, “what is revenue by store for the top five stores”, “what is our average order value”. One aggregate, one or two joins, done. The hard tier is multi-step composition: “what is the month-over-month revenue growth rate for each month” needs a CTE that aggregates to months and then a LAG window function that compares each to the one before. “What is each store’s share of total company revenue” needs a window SUM over a grouped aggregate. “How many customers spent more in the last 90 days than in the 90 days before that” needs two non-overlapping date windows, per-customer conditional aggregation, and a comparison. These are not trick questions, and we wrote them so that the rules stated in the context would pin down a single answer (we did not fully succeed on the first try, as you will see). We will come back later to their purpose.

For each question we render the context at a series of levels supposed to vary the derivation effort, and hand it, with the question, to a model. The model writes SQL, we run it against the actual database, and compare the result to the known answer. We do this for three models spanning a wide capability range (Haiku 4.5, Sonnet 4.6, Opus 4.8), repeat each cell three times to catch flakiness, and count a model that writes invalid SQL as wrong. That makes for a total of 2,484 runs.

The...