Agent in the middle - by Alex Nork - Alex's Substack

Alex's Substack

SubscribeSign in

Agent in the middle<br>How we made an AI speech pipeline feel like one continuous thing

Alex Nork<br>Jul 24, 2026

Share

Partway through building voice mode, OpenAI released GPT-Live, a native speech-to-speech model. Audio in, audio out, one model, full duplex. It listens and answers the way a person does, one continuous thing, and we explored whether we could build on it.<br>A speech-to-speech model makes the voice model the agent. It can call a function, but everything our assistant is: tools, memory, skills, approval flows, would have to be rebuilt inside the provider’s runtime. And the model that speaks is the model that thinks: there’s no routing a hard question to a stronger brain. Maybe that changes someday. Today, for an assistant whose whole value is the agent loop, that trade kills the product before it ships.<br>So we kept the classic cascade, the one that’s always felt robotic: transcribe the speech, run the full agent loop, synthesize the reply. Three stages bolted together, and every joint between them is a seam you can hear as dead air. We spent the engineering on hiding those seams. The goal, written down early and returned to often, was to make voice mode feel natural.<br>How it started

Voice began as a feature of the existing chat surface. Press a button, talk instead of type, watch the same transcript fill in. It worked on the first try, and it felt wrong in a way that took a few days to name: watching both sides of the conversation render as text while you talk turns the conversation into a deposition. You stop talking to something and start reading a court record of yourself. The transcript was breaking it.<br>That observation ended voice as a feature of our chat interface. We built it a room instead. Start a conversation and the assistant grows to fill the screen: the walls take its color, its two cartoon eyes look back at you, a “Listening” label sits beneath them, and that’s the whole interface. The surface came together in days. It looked alive.

But it didn’t sound alive. Underneath the eyes, the interaction model was a walkie-talkie: you talk, it pauses, it thinks, eventually it talks back. Every silence reminded you there was machinery back there. The gap between how it looked and how it felt lived entirely in the turn-taking.<br>We could have shipped it that way. It worked, and the launch date was close. We decided to spend another week building, because voice gets one first impression. Someone who tries it once and feels the walkie-talkie closes the window, and no changelog entry brings them back to check whether it got better.<br>The pipeline

A single WebSocket carries the whole conversation between the browser and the server.

Each turn routes through the same agent loop that text chat uses, so voice inherits tools, memory, and skills for free. That’s the naive version, and the naive version feels mechanical. A handful of mechanisms make it feel alive.<br>Speculative launch

The longest pause in a cascade sits between the user’s last word and the model’s first token. We start early to hide it: the moment the voice detector hears trailing silence, the agent turn dispatches, before we’ve confirmed the person is actually done talking. There is no separate confirmation step to wait for: the same model call that begins the reply judges, from the wording, whether the person was actually finished. If it turns out they were only pausing mid-thought, the whole turn rolls back, persisted message included, and the listening window quietly extends. You pause, it waits. You finish, it’s already answering. This overlap is the single biggest latency win in the system.<br>The mouth and the brain

Every turn goes first to a fast, cheap “front” model, and the first tokens of its reply are a verdict.

Three verdicts are possible.<br>Simple turn: the front model just answers. Its output streams straight to text-to-speech, and first audio arrives fast because a small model is doing the talking.<br>Hard turn: the front model escalates. It says one short phrase out loud, usually naming what it’s about to do, like “let me check your calendar.” It then hands the turn to the strong model, which runs it in the background while that phrase plays. The mouth covers for the brain, so the caller never hears the big model’s think time as silence.<br>Not a turn at all: if the wording says the person isn’t finished, a trailing “and...” or a thought left hanging, the front model stays quiet and we keep listening.<br>The front model has no tools, but it knows what the strong model can do, so it hands off hard requests instead of refusing them or making things up.<br>Floor-holders

Even with the front model in place, there are stretches where nothing is being said: usually a turn running long on tool calls. The assistant fills those gaps out loud. Some of it is a simple acknowledgment, “one sec, let me look.” But on long turns it gets better than filler: the main...