LLM Gateway vs Open Router Benchmark · GitHub

/" data-turbo-transient="true" />

Skip to content

-->

Search Gists

Search Gists

Sign in

Sign up

You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

{{ message }}

Instantly share code, notes, and snippets.

smakosh/benchmark.md

Last active<br>July 22, 2026 16:31

Show Gist options

Download ZIP

Star

(0)

You must be signed in to star a gist

Fork

(0)

You must be signed in to fork a gist

Embed

Select an option

Embed<br>Embed this gist in your website.

Share<br>Copy sharable link for this gist.

Clone via HTTPS<br>Clone using the web URL.

No results found

Learn more about clone URLs

Clone this repository at &lt;script src=&quot;https://gist.github.com/smakosh/0ca360230fb267d375b91dbd50548b67.js&quot;&gt;&lt;/script&gt;

" readonly="readonly" data-autoselect="true" data-target="primer-text-field.inputElement " aria-describedby="validation-6bf5cd62-3964-414b-ad11-4b2d7d62f38c" class="form-control FormControl-monospace FormControl-input FormControl-small rounded-left-0 rounded-right-0 border-right-0" type="text" name="gist-share-url-sized-down" />

Save smakosh/0ca360230fb267d375b91dbd50548b67 to your computer and use it in GitHub Desktop.

Embed

Select an option

Embed<br>Embed this gist in your website.

Share<br>Copy sharable link for this gist.

Clone via HTTPS<br>Clone using the web URL.

No results found

Learn more about clone URLs

Clone this repository at &lt;script src=&quot;https://gist.github.com/smakosh/0ca360230fb267d375b91dbd50548b67.js&quot;&gt;&lt;/script&gt;

" readonly="readonly" data-autoselect="true" data-target="primer-text-field.inputElement " aria-describedby="validation-724db612-fbdd-4d43-8e4d-877ca71db2ae" class="form-control FormControl-monospace FormControl-input FormControl-small rounded-left-0 rounded-right-0 border-right-0" type="text" name="gist-share-url-original" />

Save smakosh/0ca360230fb267d375b91dbd50548b67 to your computer and use it in GitHub Desktop.

Download ZIP

LLM Gateway vs Open Router Benchmark

Raw

benchmark.md

LLM Gateway vs. OpenRouter — TTFT benchmark

Phase-by-phase latency comparison using rbadillap/ai-gateways-benchmark (Python stdlib only, raw sockets — it splits every request into DNS / TCP / TLS / TTFB / TTFT so connection cost and gateway overhead don't get conflated).

Method : claude-haiku-4.5 on both gateways · same prompt · max_tokens=16 · 75 cold + 75 warm runs, interleaved round-robin so time-of-day drift hits both equally · cold = fresh connection (DNS+TCP+TLS, new TLS context, no session resumption) · warm = second request on an already-open socket · measured 2026-07-22 from a single residential connection · zero errors across all 300 requests .

Results

TTFT end-to-end (medians)<br>TTFB<br>TTFT (cold)<br>TTFT (warm)

LLM Gateway<br>201ms<br>906ms<br>814ms

OpenRouter (direct)<br>1369ms<br>1392ms<br>1232ms

TTFT (cold) is end-to-end: DNS + TCP + TLS + time to first content token — what a short-lived process pays. TTFT (warm) is the connection-pool case.

Full phase breakdown (medians, ms)

Gateway<br>DNS<br>TCP<br>TLS<br>TTFB<br>TTFT<br>Cold e2e TTFT<br>Warm TTFB<br>Warm TTFT

LLM Gateway<br>3.2<br>20.9<br>40.2<br>200.9<br>829.5<br>906.1<br>175.7<br>813.8

OpenRouter<br>3.4<br>7.2<br>12.7<br>1369.4<br>1369.8<br>1392.0<br>1228.7<br>1232.1

Spread — median (p10–p90), ms

LLM Gateway<br>OpenRouter

Cold TTFB<br>201 (194–240)<br>1369 (979–1643)

Cold e2e TTFT<br>906 (803–1380)<br>1392 (1002–1675)

Warm TTFB<br>176 (171–193)<br>1229 (924–1500)

Warm TTFT<br>814 (673–1379)<br>1232 (924–1501)

Observations

LLM Gateway is ~35% faster to first token cold (906 vs 1392ms) and ~34% faster warm (814 vs 1232ms) at the median. Its p90 cold e2e TTFT (1380ms) is still below OpenRouter's median.

The TTFB gap is architectural, not just network. LLM Gateway starts the response stream in ~200ms, before the first upstream token, and its warm TTFB is extremely stable (171–193ms p10–p90). OpenRouter's first byte arrives together with its first token (TTFB ≈ TTFT on every run) — it holds the response until the upstream answers. TTFT is the honest headline number; TTFB mostly tells you who streams headers early.

Different upstreams. OpenRouter picks its upstream per request (a test request was served via Amazon Bedrock); LLM Gateway's runs were pinned to Anthropic's API. Both are each gateway's default behavior for this model.

Vantage point matters. As the benchmark's README says, results are a property of where you measure from, not a global ranking. OpenRouter's edge terminated in Copenhagen for these runs (cf-ray: …-CPH); request receipts were captured per run.

Reproduce it

git clone https://github.com/rbadillap/ai-gateways-benchmark<br>cd ai-gateways-benchmark<br># config.json:<br>"runs_cold": 75,<br>"runs_warm": 75,<br>"prompt": "Reply with the single word: pong",<br>"max_tokens": 16,<br>"gateways": [<br>"name": "llmgateway",<br>"host": "api.llmgateway.io",<br>"path":...