Agents don't need smarter models, they need a map

Agents don't need smarter models, they need a map

Horia Cristescu · July 2026 · karto

Watching agents fail

The method described here was discovered watching computer-use agents fail. Their traces showed two patterns, over and over. [1] See but don't understand: the app's true state hangs on a minuscule indicator, a red dot, a greyed icon, while the model attends to the whole screen with flat salience. It has the pixels but not the significance, because knowing which pixel matters in this app is not something general vision training can supply. [2] Understand but can't act: the model reads the situation correctly, but the app demands its own ritual, the click that only registers after an animation, the form that drops input unless you tab out. Perception quirks and action quirks. Every app and every website has both, and no two share them. Models do absorb the common conventions, but the long tail of quirks is app-specific and changes with every release, which makes it a poor target for training and a good target for writing down. The knowledge has to live somewhere local.

But the current architecture has nowhere to put it. Today's agent maps screenshot to action in one jump, and a one-jump mapping has no place where "in this app, the red dot means unsaved" could be written down, checked, or corrected. It can only be dissolved into the prompt or the weights. A State Space Policy (SSP) inserts a middle term: screenshot to named state to action. The named state is the place where local knowledge lives, and it is where failures get an address. In its smallest form it looks like this:

clean-editor {modal: none, save: clean, focus: editor}<br>-> proceed-with-task<br>unsaved-editor {modal: none, save: unsaved, focus: editor}<br>-> save-before-navigation<br>blocked-by-modal {modal: blocking, focus: dialog}<br>-> resolve-modal-first<br>save-failed {modal: none, save: failed, focus: editor}<br>-> retry-save-then-verify<br>Each line is a named situation, the handful of feature values that identify it, and the action that works there. clean-editor and unsaved-editor differ by a single value, and that value flips the correct action, which is why the save indicator gets a place in the vocabulary while the rest of the screen does not. The app's local state was never "in the screenshot" in any usable way. It had to be named.

State Space Policies are a method for building that representation: a named state map between perception and action. An SSP turns a domain into a small coordinate system: actions, the states that determine those actions, and the features that discriminate the states, induced from actual traces of the domain rather than from a prior ontology. The vocabulary does not need to recover the domain's true latent geometry. It needs to be something a human can inspect, argue with, revise, and eventually run. Computer-use is the running example because its failures are so visible, but nothing in the method is about screens: the same structure applies wherever an agent acts against an environment that pushes back, coding agents and their test suites, ops agents and their incident traces, tool-use pipelines of any kind. Wherever there are traces of actions with outcomes, the induction below applies.

The waist

Underneath the tooling is a factorization. Any problem of situated competence is a huge mapping from observation histories to actions. An SSP factors it through a narrow middle layer: recognition maps many messy inputs onto few named states, and policy maps few named states onto appropriate actions. Each half can be learned, debugged, and argued about on its own. The end-to-end mapping has all the same structure implicitly, somewhere in the weights, but you cannot see where the recognition ends and the decision begins, and you cannot control which shortcuts the model takes between them.

The bet is that if the state chunking is done well, policy becomes easy. Look back at the four-line map in the opening: once the screen is recognized as unsaved-editor, choosing the action is a lookup. All the difficulty moved into the recognition, and it was paid when the vocabulary was designed. Most of the method's work is thinking carefully about state.

This is what a doctor does, and medicine built the apparatus explicitly. A differential diagnosis is a small state space: a candidate set of conditions, a set of discriminating findings, and a treatment attached to each condition. The doctor does not perceive the patient exhaustively and does not compare all treatments. They narrow the search space with a few targeted questions, each answer ruling out candidate conditions, until one remains and its treatment follows. The differential that takes minutes at the bedside took medicine centuries to build: naming the diseases, learning which findings tell them apart, attaching a treatment to each. Every consultation spends that accumulated work without repeating it. An SSP is the same kind...