The Three Moments Your AI Gateway Can Ruin · Highflame

Highflame Identity is now open source: agent identity on open standards. Read the launch →

Book a Demo

Book a Demo

← All articles<br>Put a fast model behind the wrong gateway and your user stares at an empty chat bubble for a second and a half. The model already sent its first token; the gateway is holding it. We measured exactly that happening.

If you run models in production, you end up with a gateway eventually. A good one routes every provider through a single URL, keeps API keys off developer laptops, meters spend per team, and scans traffic before it leaves. The catch is that it sits inside every request your users make. So before you pick one, you want to know what it does to the moments they actually feel.

In Part 1 we measured raw gateway cost against an instant backend. Nobody ships an instant backend, so this time it behaves like a real model: first token around 300 ms, a two-second turn, tokens streamed one at a time. In front of it: Highflame (Rust), Bifrost (Go), and LiteLLM (Python).

Three moments decide the experience:

The first token. Does the app still feel instant?

Peak hour. When everyone talks at once, do calls come back?

The tool call. What does an agent pay per action?

Moment one: the first token

The silence before the first token is where a user decides whether the app hung. The model owns almost all of that silence: with no gateway anywhere in sight, it takes about 300 ms to produce its first token. The only question for a gateway is what it adds on top of that.

Highflame adds about 2 ms . That is the whole toll, measured at ten chats where noise is smallest, and it is paid once at the front: the completed answer lands within single-digit milliseconds of the direct stream at every concurrency we ran, so no cost accumulates per token. At a hundred simultaneous chats the first token through Highflame lands in 307 ms , indistinguishable from hitting the model with no gateway at all.

One model turn, three gateways animation of the measured streaming behavior · fifty concurrent chats

The model is on the right, the user on the left. Highflame passes every token<br>through as it arrives. Bifrost holds the stream and releases it whole. LiteLLM<br>streams, but each token pays a toll that compounds with load: ~350 ms of added<br>wait at the fifty chats shown here, six seconds at a hundred (charted below).

What the gateway adds to the first token added wait at 100 concurrent chats · the model's own ~300 ms excluded<br>Highflame +2 ms measured at ten chats; below noise here<br>Bifrost +1.3 s holds the stream until the end<br>LiteLLM +6.0 s per-token cost compounds

Every pixel of bar here is gateway overhead; the model's own wait is excluded.<br>Highflame adds about 2 ms, a sliver at this scale. LiteLLM adds six seconds at<br>this load, and Bifrost adds 1.3 seconds by holding the stream until the answer<br>is done.

The failures are worth naming, because you would never catch them on a spec sheet:

Bifrost buffers. In its stock configuration it forwards the stream but holds every byte until the model finishes, so the first token shows up together with the last one. That is 1.3 seconds of gateway-added wait on every turn, and the UI reads as frozen even though tokens were flowing the whole time.

LiteLLM compounds. It streams properly at ten chats, but its per-token work stacks up under load. At fifty concurrent chats it adds about 350 ms to the first token; at a hundred, the median first token slips past six seconds , and the streamed answer that takes 1.6 seconds direct takes 7.7 through it.

First token as chats pile up median time to first streamed token · same simulated model<br>10 chats 308 ms<br>358 ms

50 chats 304 ms<br>660 ms

100 chats 307 ms<br>6320 ms

Highflame LiteLLM the model alone: ~300 ms

Everything past the dashed line is wait the gateway added. Highflame's first token<br>stays on the model's own line as load grows. LiteLLM's slides from 358 ms to 6.3<br>seconds at the median, with a p99 of 9 s at a hundred chats, and the<br>whole answer stretches from 1.6 to 7.7 seconds.

Moment two: everyone talks at once

Real conversations hold connections open. Each request waits about two seconds for its turn to finish, so a busy afternoon (agent fleets fanning out, a CI run, the whole company inside the same hour) turns into thousands of sockets held open at once.

Five minutes of peak hour, compressed 5,000 conversations held open · answered, dropped, and the memory bill

Every dot is a held conversation waiting on a two-second model turn. Highflame<br>and Bifrost answer everything; Highflame does it on half of Bifrost's memory.<br>LiteLLM's tank empties out as seven in ten of its calls never come back.

We held 5,000 conversations open simultaneously for five minutes, every one waiting on a two-second model turn. Highflame answered all 572,563 requests it was offered, a true 100% , with a p99 of 3.3 seconds , and it carried the whole rush in 733 MB of memory. Bifrost...