The Production AI Stack: A Reference Architecture for Real-Time AI Systems<br>Loading<br>Preparing your content

Loading<br>Preparing your content

Quick Links

Integrations

Legal

Built with ❤️ by the Moss team

Back to Blog

A founder built an AI support agent for his ecommerce store using what has become the standard modern AI stack: a large language model, a cloud vector database containing the company's help center, and a small set of tools for common actions like looking up orders and initiating returns. The system worked well. Customers could ask questions, check the status of an order, or begin a return without ever needing to speak with a human.

The problem wasn't correctness. It was latency.

Every interaction followed the same execution path. A customer asked a question, the agent queried the vector database for relevant context, waited several hundred milliseconds for the results to return, and only then could the model begin generating a response. Nothing in the architecture was technically broken, but the cumulative delay was obvious enough that conversations felt slower than they should.

Like many teams building production AI systems, he optimized for responsiveness by removing retrieval altogether.

Instead of retrieving context on every turn, he embedded everything directly into the system prompt. The help center, shipping policies, return rules, pricing information, FAQs, and any other information a customer might reasonably ask about all became part of the prompt that accompanied every request. It eliminated an external dependency, reduced latency, and simplified the architecture.

For a while, it looked like the right tradeoff.

As conversations became longer, however, the architecture started failing in more subtle ways. Every message added another layer of conversational history to a context window that was already carrying the company's knowledge base, forcing the model to reason over an increasingly large body of information. As the available context filled up, retrieval accuracy was effectively replaced by probabilistic recall. Shipping questions began pulling in return policies, discounts from unrelated products appeared in responses, and facts that had been clear at the beginning of the conversation became less reliable over time.

The most frustrating part was that these failures rarely appeared during testing. Short benchmark prompts continued to perform well, while the conversations that actually mattered in production were the ones that exposed the weaknesses of the architecture. Customers asked follow up questions, referred back to earlier messages, changed their minds halfway through a workflow, and expected the system to maintain consistency across dozens of conversational turns. Those were precisely the interactions where the model became least reliable.

This pattern appears across every category of production AI application, from customer support agents and enterprise copilots to voice AI platforms and conversational search systems. Teams often assume they're dealing with a prompting problem or a model quality problem, when in reality they're running into architectural limits. Large prompts eventually become expensive to process, context windows inevitably become saturated, and relying on a language model to remember everything produces systems that become less predictable as conversations grow longer.

The Production AI Stack

Every production AI system, whether it's powering a voice AI platform, an enterprise copilot, or a conversational search experience, is ultimately composed of the same seven architectural layers:

Models determine which foundation model is responsible for each task.

Inference controls where and how token generation happens.

Search retrieves the knowledge required to answer each request.

Memory persists information about users and previous interactions.

Sessions maintain conversational state across multiple requests.

Orchestration coordinates the execution of every component throughout the conversation.

Deployment determines where each layer runs, whether in the browser, at the edge, on device, or in the cloud.

Each of these layers can be optimized independently, but production AI systems are rarely limited by any single component. Performance, latency, reliability, and cost emerge from the interactions between them. Understanding those tradeoffs requires looking at the entire architecture rather than any individual layer, which is where we'll begin.

Start With the Latency Budget

Every production AI system begins with the same constraint: latency.

Before deciding which model to use, how to structure retrieval, or where to run inference, you need to understand how much latency your users will actually tolerate. Every interactive product has a latency budget, and once that budget is exceeded, no amount of model quality can recover the user experience.

Human computer interaction research established these thresholds decades before large language...