Managing a small local AI budget — millfolio ← Blog In the first post I described millfolio’s split: a frontier model writes a small program over an aliased schema, and a local model reads your actual files on-device. I ended with a promise to write about the unglamorous part — running the local model in batch mode, with priorities, so your laptop stays usable. This is that post.

The problem: the model doesn’t scale with your records

A couple of years of bank and card statements is thousands of transactions. The local model is good at reading one of them — “is this a subscription?” is exactly the kind of targeted question it answers well. What it cannot do is answer that question thousands of times, per user question, on a laptop GPU. Even at a fraction of a second per record, a question like “what did I spend on subscriptions last year?” would turn into minutes of inference — and the next question would do all that work again.

So the design rule became: the model’s judgment is a thing you compute once and store, not a thing you run at question time.

The answer: tags, in three flavors

Every transaction gets tags at index time, from a plain-text rule file you can open and edit (categories.txt — one rule per line, the file is the source of truth). The three flavors are ordered by cost:

String tags — the cheap, common case. A tag is a list of case-insensitive substring keywords, with optional excludes that veto a match:

phone = verizon, at&t, t-mobile, mint mobile<br>groceries = whole foods, trader joe, safeway, costco<br>Pure string matching, effectively free, and it covers most of a real spending history — merchant strings are repetitive.

Reference tags — tags built out of other tags. A rule can reference @tag names instead of keywords, so you can compose groups without repeating anybody’s keyword list:

food = @groceries, @restaurant<br>Still deterministic and free — a reference tag fires when a referenced tag is already on the row. Reference cycles are harmless (tags are only ever added, so evaluation just converges), and a reference is allowed to point at an AI tag, which makes the third flavor composable too.

AI tags — the fuzzy tail no keyword list captures. The rule is a yes/no question, and the on-device model is the judge:

subscriptions : is this a recurring subscription charge?<br>This is the only flavor that costs inference — which is exactly why it’s a stored tag and not a query-time call.

At question time, all three look identical: the generated program filters on .tags, and the answer over thousands of records is a string comparison, not a model call. There’s also a privacy dividend: the frontier model is told the tag names and scope notes (so it writes programs that filter on them), but never the keywords — your actual merchant strings stay on the device.

Batches, and knowing when to nap

AI tags still have to be computed once, and “once” means a backfill over every existing record plus whatever arrives later. That work runs through a single on-device work orchestrator: a disk-backed queue that survives restarts, draining one job at a time — indexing and AI-tag backfill can never fight each other for the engine, and both yield to an interactive question.

Inside a backfill job, the classification is batched and deduplicated:

Many descriptions go to the model in one prompt , and it answers 1: yes, 2: no, … — one inference call classifies a whole slice.

Exact-duplicate descriptions are collapsed before the model sees them : a monthly charge that appears 24 times is classified once and the verdict fans out. On real statements this saves a large fraction of the work (the Stats page keeps a running counter of records classified vs. records covered).

A small ledger records how far each rule has been materialized, keyed on a monotonic insertion generation — so a finished tag never re-runs over old records; new records get picked up incrementally.

Between slices, the backfiller naps — and the nap length is the priority setting: high is a 100ms breath, medium is 1.2s, low is 5s. That one knob is how you decide whether the machine is “get it done” or “I’m working, take all night.” You can also pause the background processing indefinitely.

How fast/slow is it?

The following stats were generated by running mill run https://raw.githubusercontent.com/millfolio/vault/5915bf52f17b9baf2c4fe71695dae03e05a9ee16/privacy-box/eval/local_classifier_eval.mojo on a Mac M2 16GB. (The program prints only aggregates — no merchant strings or amounts — and runs on any millfolio install; the live progress lines it streams while running need millfolio ≥ 0.4.49.)

The raw output is below

Model: Qwen/Qwen2.5-3B-Instruct<br>Question: "For each snippet: answer 'yes' if it is a grocery store or supermarket purchase (food shopping), answer 'no' for everything else (dining out, restaurants, coffee shops, and any non-grocery charge). Every answer must be exactly 'yes' or 'no' — never 'none', never anything else." vs the keyword...