Not everything should cost a token: the case for deterministic AI | Vybe BlogUse casesIntegrationsBlog<br>LoginBook demoRequest access

Best PracticesNot everything should cost a token: the case for deterministic AI<br>Running every task through an AI model burns tokens and bloats context. Learn what not to tokenize, and why the best agent systems push deterministic work down to the app layer.<br>July 6, 2026<br>9 min read

Not Everything Should Cost a Token: The Case for Deterministic AI

A team I talked to recently wired up an agent to do something simple: pull a metrics API every morning, reshape the JSON, and drop the result into a table. Clean idea. It worked on day one.

Then the invoice came in. Every morning, that "simple" job was loading a few thousand tokens of raw JSON into a context window, asking a language model to reformat data it did not need to reason about, and paying for the privilege. The reformatting was occasionally wrong, too, because a probabilistic model was doing a job that a five-line script does the same way every time. The context window was so stuffed with stale data that the actual reasoning the agent was supposed to do got worse.

That trap catches almost everyone right now. Teams treat the language model as a universal runtime. If you can describe a task in a sentence, you prompt it. But a model is not a cron job and it is not a database. It is a reasoning engine you are renting by the token, and the fastest way to blow up your bill and degrade your output is to make it do work that never needed a brain in the first place.

The skill worth learning is knowing what not to tokenize.

Why "everything is a prompt" feels right and costs so much

The appeal is obvious. Prompting is fast. You skip the schema design, the endpoint, the deploy. You just ask. For a one-off, that is genuinely the right call.

The problem shows up when the one-off becomes a recurring job. Every deterministic task you route through a model inherits three properties it should never have had: it becomes non-deterministic, it becomes slow, and it becomes metered. You now pay, per run, for work that a scheduled function would do for free and never get wrong.

Two symptoms follow, and they compound each other.

The first is cost that scales with the wrong thing. Your token spend should track the value of the judgment you are asking for. Instead it tracks the volume of mechanical work you have shoved through the model. Pull ten APIs a day instead of one and your bill grows tenfold, even though none of that pulling required intelligence.

The second is context bloat. To "process" data, teams read it into the context window. But the context window is finite and it is where reasoning happens. Fill it with raw records and you crowd out the thing the model is actually good at. Quality drops, and because most providers meter input tokens, the cost of a long context climbs at the same time. You pay more to get worse answers. That is the part that stings.

The distinction that fixes it: probabilistic vs. deterministic work

Here is the mental model I keep coming back to. Sort every task into one of two buckets before you decide where it runs.

Belongs in the Agent (probabilistic)Belongs in the App (deterministic)Judgment and interpretationScheduled and recurring tasksDrafting, summarizing, classifyingAPI calls and data transformsDeciding what to do and whether something mattersStoring, retrieving, and querying dataHandling ambiguity and edge casesAnything that must be exact and repeatable every single run

The line is simple. If a task needs judgment, it belongs with the agent. If it needs to be exact and repeatable, it belongs in an app. Determinism is a feature, not a limitation. A scheduled job that runs identically every morning is something you want to be boring and predictable.

And here is the anchor line for the whole idea: you build the app with tokens once, then you stop paying tokens every time it runs. The intelligence goes into creating the machine. The machine itself runs for free.

"Learn what not to tokenize" is a discipline you practice on every task, not a setting you toggle once.

The one everyone gets wrong: memory notes as a database

There is a specific version of this mistake that is worth calling out on its own, because I see smart people do it constantly.

People use their agent's memory or notes as a database.

They start dropping structured data into notes: records, metrics, pipeline stages, customer lists, inventory counts. It feels natural because the note is right there and the agent already reads it. But notes are the wrong tool for structured data, and the reasons pile up fast. There is no schema, so nothing enforces consistency. There is no query, so the agent reads the entire note into context every time it needs one field, which is both slow and metered. And notes degrade as they grow, because a wall of semi-structured text is exactly what a real table exists to replace.

Now here is the part...