, so they MUST live in the<br>Vercel header config. -->

(react-helmet-async) and baked<br>into static HTML at build time by scripts/prerender.mjs — so each route,<br>including localized landing pages, ships a single correct set with no<br>duplicates and the page title injected per route. -->

Stop Sending Giant System Prompts: Treat LLM Tokens Like a Scarce Resource | Qolca Blog

Stop Sending Giant System Prompts: Treat LLM Tokens Like a Scarce Resource

Qolca Team · 2026-07-16 · 11 min read

Most AI products overpay because they stuff everything into one enormous prompt and send it to the most expensive model on every request. The fix is not a cheaper vendor. It is treating tokens like a scarce resource: decompose the flow, retrieve instead of context-stuffing, cache the static parts, and reserve the expensive model for the one step that needs it.

There is a failure mode in almost every AI product that is quietly bleeding money: one giant system prompt, stuffed with every rule, every few-shot example, every edge case, and every scrap of company knowledge, sent to the most expensive model available on every single request. It works beautifully in the demo. Then real traffic arrives, the bill arrives with it, and latency creeps up because the model is re-reading a small novel before it answers 'what are your hours?'. The instinct is to go shopping for a cheaper vendor. That is the wrong move. The problem is not the price of the model — it is that you are spending tokens like they are free. They are not. This is a piece about treating them like the scarce, metered resource they actually are.

The Hidden Tax on the One-Big-Prompt Design

Every token in your prompt is paid for on every request. A frontier-tier model costs on the order of five to fifteen dollars per million output tokens and a few dollars per million input tokens; a small, fast model in the Haiku class costs roughly a third to a fifth of that. Those numbers sound tiny until you do the multiplication. Suppose your system prompt is 4,000 tokens — a modest size once you have pasted in your product catalog, your tone guidelines, and a dozen examples. In a ten-turn conversation, the model re-reads that prompt on every turn, so you have paid for 40,000 input tokens just to restate instructions the model already absorbed on turn one. Now multiply by every conversation, every day. The waste is not dramatic per request; it is dramatic in aggregate, and it scales linearly with your success.

Cost is only half the tax. The other half is latency, and latency is a product feature. A model cannot emit its first token until it has read your entire prompt, and a bigger prompt means more to read and more to reason over before anything appears on screen. A 200-token routing decision and a 6,000-token everything-prompt feel completely different to a user waiting for a reply. When teams complain that their chatbot feels sluggish, the culprit is very often not the model — it is the wall of context they force it to wade through on every turn. You are paying twice for the same mistake: once in dollars, once in seconds.

Five Patterns That Treat Tokens as Scarce

The mental shift is simple to state and surprisingly hard to adopt: stop asking "what could the model possibly need to know?" and start asking "what does this specific step actually require?" Once you hold tokens to that standard, five patterns fall out naturally. None of them are exotic. They are just discipline applied to a resource most teams treat as infinite.

Decompose the flow into steps, so each model call carries only the context that step needs<br>Retrieve instead of context-stuffing, pulling in the two relevant paragraphs rather than the whole manual<br>Cache the static prefix, so the parts of your prompt that never change are not re-billed at full price<br>Classify and route cheaply, using a small fast model to decide what happens next<br>Reserve the expensive model for the one step that genuinely needs its intelligence

Decompose the Flow Into Steps

A single request to "handle this customer message" is really several smaller jobs wearing a trench coat: figure out what language they are speaking, work out what they want, gather any relevant history, and then write a good reply. Bundled into one prompt, every one of those sub-jobs pays for the full context and the full model. Split apart, each becomes a small, cheap, focused call. Language detection needs the message and nothing else. Intent classification needs the message and maybe a line of history. Only the final reply needs the persona, the tone, and the product knowledge. Decomposition is the master pattern — the other four are things you can only do once the flow is broken into steps.

Retrieve Instead of Context-Stuffing

The most common form of token waste is pasting an entire knowledge base into the prompt on the theory that the model "might need it." It almost never needs all of it. If a customer asks about your refund window, the model needs your refund...