leteo/docs/nothing-worth-tuning.md at main · asanabrial/leteo · GitHub
//blob/show" data-turbo-transient="true" />
Skip to content
Type / to search
Sign in<br>Sign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
asanabrial
leteo
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
FilesExpand file tree
main
/nothing-worth-tuning.md
Copy path
Blame<br>More file actions
Blame<br>More file actions
Latest commit
History<br>History<br>History
100 lines (54 loc) · 14.6 KB
main
/nothing-worth-tuning.md
Copy path
Top
File metadata and controls<br>Preview
Code
Blame
100 lines (54 loc) · 14.6 KB
Raw<br>Copy raw file<br>Download raw file
OutlineEdit and raw actions
I tried to teach my retrieval to say "I don't know", and there was nothing worth tuning
I maintain a small local-first memory tool for coding agents. One Rust binary, one SQLite database, full-text search over what previous sessions wrote down. The part this post is about is twenty-five lines long.
Search runs in three stages. Every word must match; failing that, all but one word; failing that — the stage in question — any of the words, ranked, keeping only what stands out relative to the rest of the result set. The floor is a ratio against the median bm25 of the sample. FTS5 scores are negative and lower is better, so multiplying the median by 1.2 makes the bar stricter, not looser. That trips everybody, including me, twice.
That third stage did a lot. Over 277 real prompts — the shape an agent actually types into this thing, not citation-shaped queries — the first two stages together came back empty 80.5% of the time. Adding the third took that to 7.2%.
And it has a price, which I measured the same day and wrote down next to the rest: asked a question from a different project, with the read scoped to this one as it always is, it still speaks 67.9% of the time. A relevance floor is dimensionless. It knows what an ordinary match looks like for this query. It does not know whether the store has the answer.
The push
I posted the 80.5%-to-7.2% number on Reddit, and somebody replied that it measures coverage, not usefulness. Which is right. A stage that nearly always finds something drives its own empty rate towards zero by construction, and the 67.9% is the same fact seen from the other side. Whatever comes back goes into an agent's context, and it is read as relevant whether or not it is. The answers are marked partial and the wording never claims they matched, which I think helps less than I would like it to.
So the honest follow-up isn't to defend the number. It's: is 67.9% a badly chosen operating point? Maybe an absolute bm25 cut, sitting next to the relative one, would let the stage stay quiet when there is nothing there.
That was the hypothesis. What came back was far smaller than it needed to be.
Setup
A copy of my real store: 4,448 memories across 18 projects. Reads are scoped to one project by default, and every query in this post was scoped to leteo — so the corpus each question actually ran against is the 359 memories filed under it, not all 4,448. That is worth holding on to when you get to the part about common words.
Two sets of real prompts, both asked against leteo:
home — 296 prompts recorded while working on leteo itself
control — 399 prompts from two other projects
If bm25 carries any information about whether an answer exists, home should speak more often than control at some threshold, on some rule. That is the whole test.
One detail mattered more than it sounds: the probe runs the product's own query builder. Not a copy of it — the actual SQL builder, weights, sample depth, floors and the ranking predicate, re-exported behind a Cargo feature that is off by default. I had been caught by this once already. An earlier harness kept its own copy of the ranking query and its own idea of what counts as a letter — described in its comment as "the partition the binary uses", which it was not. It treated Greek, Cyrillic and the Spanish ordinals (ª, º) as separators where the binary treats them as letters. Over 4,055 memories that one difference moved top-1 title accuracy from 78.3% to 80.1%. The harness was measuring a search nobody runs, which is the exact trap its own comment warned about.
The related rule of thumb, from a separate mess: timing does not transfer between SQLite builds; ranking does; query construction does not transfer at all.
Result
Percentage of prompts where the stage returned anything. The probe drives that stage directly rather than running a whole search, so these are the stage's own rates, not what a user would see end to end — in the product it only runs when the two stricter stages have already come back empty:
= D, 0 to 3.25 97.3→73.6...