The same LLM is 8× slower to first token depending on who serves it — Dynoyard ← Engineering We serve GLM-5.2 to teams building agents. Same open-weight model, same OpenAI-compatible API — but we route it across more than one backend, and while swapping one in we found something worth writing down: the backend you pick changes time-to-first-token by 8×, and one of them wasn’t really streaming at all.

The setup

Two backends for the identical model. A dedicated inference host, and a major cloud provider’s managed OpenAI-compatible endpoint — cheaper per token, a very large context window, tempting. Both accept a stream: true request and return an SSE stream. On paper, interchangeable.

What we measured

We fire 40 cache-busted requests at each, streaming, and record per request: time-to-first-token, total time, and — the important one — whether tokens actually arrive incrementally or all at once at the end. Numbers (p50, 400-token outputs):

Dedicated hostCloud managed endpointTTFT1.2s 10.0s Total5.2s10.4sOutput tok/s per stream7738Actually streamed?yes (167 chunks)no — 1 burst at the end<br>The cheap endpoint generates the entire response, sits on it for ~10 seconds emitting nothing, then dumps all the tokens at once. It’s stream: true-compliant to the letter — the bytes do arrive as SSE — but there’s no incremental delivery. For a chat UI or an agent loop, that’s a dead 10-second pause every turn. The “streaming” is cosmetic.

We assumed it was distance

It’s a cloud region far from us; surely that’s the latency. So we moved it to a closer region. Result: ~18% faster, still buffered. Region wasn’t the cause — the endpoint buffers generation server-side regardless. And we confirmed it wasn’t our own gateway by running the dedicated host through the exact same path: it streamed 167 chunks cleanly. The buffering belongs to the provider.

Then the twist: caching changes everything

Cache-busted tests are worst-case. Real agent traffic is heavily cached — long stable system prompts and tool definitions on every turn, only the tail changing. So we re-ran with a warm cached prefix, and the cheap endpoint’s TTFT dropped from 10s to 2.7s , and it mostly stopped buffering. The pathological number was an artifact of cold, uncached requests that no production agent actually sends. Under realistic cached load the gap narrows a lot — though the dedicated host still won ~1.6× on every axis.

That reversal is the real lesson. If we’d benchmarked cache-busted and called it a day, we’d have written the endpoint off as unusably slow. If we’d trusted the provider’s spec sheet, we’d never have seen the buffering at all.

The takeaway

“OpenAI-compatible” tells you the request shape, nothing about behavior. Two backends serving the identical model differed 8× on the metric users feel, one faked streaming, region didn’t fix it, and cache shape flipped the whole conclusion.

Benchmark backends in your real traffic shape — cached the way you actually run — not with a cache-busted synthetic loop, and not from the provider’s spec sheet. The number that matters is per-stream effective throughput on cached prompts, measured through your own gateway. We define it as output tokens divided by (settle time − first-token time), p50, with cache-fast responses excluded — the speed a single generation actually feels, not a summed-stream aggregate that flatters you under load.

We kept the dedicated host as primary and the cloud endpoint as a failover — it has a much larger context window the other can’t match, and under warm cache it’s perfectly usable. But we only knew any of this because we benchmark every backend before it serves a request — and increasingly, on a schedule, because models drift and providers change serving stacks without telling anyone. If “did my provider quietly break streaming or tool-calls last night” is a question you’ve never been able to answer, that’s the thing we’re building.

Every model behind one OpenAI-compatible endpoint. Run your own bake-off.

Get started See pricing →

We use cookies for product analytics to improve Dynoyard. See our<br>privacy policy.

Accept Decline