Why AI Agents Should Be State Machines

Sign in

Subscribe

A monolithic agent prompt is convenient until one prompt starts doing five jobs at once: routing, extraction, tool selection, formatting, and recovery. When the agent fails, the prompt does not tell you which job failed. It only returns another opaque output.<br>I think this is where many agent systems need less prompting and more ordinary software architecture. A state machine separates the runtime into explicit states, typed transitions, validators, and recovery paths. The LLM still handles local ambiguity inside a state; the surrounding system controls what state can happen next.<br>That boundary matters because LLM calls are stochastic. If a state emits malformed output, the next state should not have to guess what went wrong. The transition should fail loudly, attach a useful error, retry only when the failure is recoverable, and route to a human or dead-letter path when it is not.

TL;DR — Key Takeaways:

A monolithic agent prompt hides routing, extraction, tool selection, formatting, and recovery inside one probabilistic call.

A state machine makes those concerns explicit: states do work, transitions choose the next step, and contracts validate handoffs.

Each state can be tested and observed separately; malformed output fails at the boundary instead of contaminating later steps.

Model routing becomes possible once work is split by state, but economics should remain secondary to reliability.

Error handling becomes architecture: deterministic validators, bounded retries, fallbacks, and dead-letter paths.

The Monolithic Prompt Failure Taxonomy<br>Entanglement. When a single prompt encodes routing logic, domain knowledge, and output formatting simultaneously, changing one dimension requires re-testing all others. I have seen agent prompts grow until nobody could change one instruction without revalidating the whole behavior surface.<br>Untestability. You cannot write a unit test for an LLM prompt that does five things at once. You can only run end-to-end evaluations and observe whether the emergent behavior stays stable. As I wrote in The End of Determinism, the stochastic nature of LLMs makes this problem structural — but a state machine gives you isolation boundaries for your evaluation suites.<br>Cost amplification. A monolithic agent sends the same context, tool schemas, and domain rules to the same model for every step. Classification, extraction, validation, and synthesis may need different levels of intelligence, but the monolith pays for the whole bundle every time.<br>Opacity. When an agent misbehaves, a monolithic prompt conflates routing, extraction, tool selection, and formatting failures into a single black box. The state machine emits structured logs at every transition. The failing state is always identifiable.

Failure ModeMonolithic promptState machine

EntanglementAll logic in one prompt — change anything, risk everythingEach state owns one concern — changes are surgical<br>UntestabilityOnly end-to-end evals possiblePer-state unit tests + contract validation<br>CostFrontier model for every callPer-state model routing — budget where possible<br>DebuggabilityFailure location unknownStructured logs at every transition<br>Error handlingAsk the same prompt to recoverRetry, fallback, dead-letter per transition

Anatomy of an Agent State Machine<br>A production agent state machine has three primitives: states , transitions , and contracts . States do work. Transitions encode control flow. Contracts enforce the shape of data at every boundary.<br>Figure 1: Monolithic prompt versus state-machine decomposition — one opaque prompt becomes discrete, testable statesfrom __future__ import annotations

from enum import Enum<br>from typing import Any

from pydantic import BaseModel, Field

class TicketIntent(str, Enum):<br>BILLING = "billing"<br>TECHNICAL = "technical"<br>GENERAL = "general"<br>UNKNOWN = "unknown"

class ClassifyOutput(BaseModel):<br>intent: TicketIntent<br>confidence: float = Field(ge=0.0, le=1.0)<br>reasoning: str

class ExtractOutput(BaseModel):<br>entities: dict[str, Any]<br>ticket_id: str | None = None<br>customer_tier: str = "standard"

class StateResult(BaseModel):<br>next_state: str<br>payload: dict[str, Any]

# --- State definitions ---

async def classify_intent(user_message: str, llm_client: Any) -> ClassifyOutput:<br>"""State 1: Classify — uses a small classification model."""<br>response = await llm_client.complete(<br>model="small-classifier-model",<br>system="Classify the user message into one of: billing, technical, general. "<br>"Return JSON with intent, confidence (0-1), and reasoning.",<br>user=user_message,<br>response_model=ClassifyOutput,<br>return response

async def extract_entities(<br>user_message: str, intent: TicketIntent, llm_client: Any<br>) -> ExtractOutput:<br>"""State 2: Extract — uses a constrained extraction model."""<br>response = await llm_client.complete(<br>model="structured-extraction-model",<br>system=f"Extract structured entities from a {intent.value} support ticket. "<br>"Return JSON with...