Benchmarking Agentgateway vs LiteLLM's Rust Mode – agentgateway | Agent Connectivity SolvedSkip to contentagentgateway has joined the Agentic AI Foundation — Learn more

Docs<br>Standalone<br>Kubernetes<br>Models<br>Blog<br>Enterprise<br>Community

@agentgatewayGet Started<br>GitHub

Benchmarking Agentgateway vs LiteLLM's Rust Mode<br>A head-to-head proxy benchmark comparing agentgateway and LiteLLM's Rust mode on throughput, latency, CPU, and memory using Fortio and a mock Anthropic backend.<br>Lin Sun<br>Aug 17, 2026<br>5 min read

← Back to blogSince I published my benchmark of agentgateway vs. LiteLLM (Part 1 and Part 2), I&rsquo;ve received quite a few questions about LiteLLM&rsquo;s newly added Rust mode. So I decided to compare its performance with agentgateway.<br>Rather than comparing features, I wanted to answer a simple question:<br>How much proxy overhead does LiteLLM&rsquo;s Rust mode introduce compared with agentgateway?

Similar to my previous benchmarks, I wanted to measure:<br>Throughput (QPS)<br>Request latency<br>CPU utilization<br>Memory usage<br>The goal is to isolate the performance overhead introduced by each proxy.<br>Enable LiteLLM&rsquo;s Rust mode

Following the LiteLLM Rust mode documentation, the recommended way to enable Rust mode is to add rust: true to the LiteLLM configuration.<br>At the time of this benchmark, Rust mode did not support the OpenAI API path I was using, so I switched to a mock Anthropic model. Here is the relevant LiteLLM configuration:<br>model_list:<br>- model_name: claude-mock # for test litellm rust<br>litellm_params:<br>model: anthropic/claude-3-5-haiku-20241022<br>api_base: http://mock-server:8081 # Anthropic-style base<br>api_key: dummy<br>rust: true

I also made sure I was using the latest LiteLLM build available for the test, LiteLLM 1.98.0, and the new agentgateway version, 1.4.0.<br>For the Anthropic benchmark, I used the /v1/messages API path, which was the supported path for the Rust mode configuration I tested.<br>I then manually verified that Rust mode was actually enabled for the requests used in the benchmark.<br>For example:<br>curl -sD - -o /dev/null http://127.0.0.1:4000/v1/messages \<br>-H "Content-Type: application/json" \<br>-d @payloads/req-anthropic-1024.json

The response included:<br>x-litellm-version: 1.98.0<br>x-litellm-rust: true

The x-litellm-rust: true header confirms that the request was handled by LiteLLM&rsquo;s Rust mode.<br>Test setup

The benchmark uses a very simple architecture. A mock LLM server immediately returns a fixed response so the benchmark measures proxy overhead rather than model inference time.<br>I used Fortio to generate traffic against each gateway.<br>fortio (bt) ──► litellm(rust:true):4000 ──┐<br>├──► mock-server (hyper-server) :8081<br>fortio (bt) ──► agentgateway:4001 ────────┘

Follow the benchmark instructions to update the LiteLLM and agentgateway configurations for Anthropic and generate the corresponding request and response payloads.<br>Max throughput benchmark

I first ran the benchmark using the default configuration while specifying the Anthropic API format:<br>./scripts/run-benchmark.sh -a anthropic

The benchmark uses:<br>API: Anthropic POST /v1/messages (both gateways)<br>LiteLLM: rust: true, 2 workers, image with 1.98.0<br>agentgateway: anthropic provider → mock<br>Load: 32 connections, max QPS for 3 seconds<br>Request payload: ~1.1 KB<br>Throughput & Latency

GatewayThroughputP50P90P99agentgateway35,502 QPS 0.863 ms 1.644 ms 1.972 ms LiteLLM (rust: true)984 QPS32.139 ms48.528 ms71.451 ms<br>Agentgateway handled over 36× more requests per second while maintaining sub-2 ms P99 latency.<br>CPU & Memory

GatewayAvg CPUPeak CPUAvg MemoryPeak Memoryagentgateway199%482%26 MiB 34 MiBLiteLLM (rust: true)69%204%2.15 GiB 2.15 GiB<br>Raw benchmark output

==> Run ID: 20260812-215223<br>==> LiteLLM workers: 2<br>==> API format: anthropic<br>==> Checking LiteLLM Rust header<br>x-litellm-rust: true<br>...<br>Running fortio to litellm at 0 QPS for 3s and 32 connections...<br>qps: 983.56qps p50: 32.139ms p90: 48.528ms p99: 71.451ms<br>Running fortio to agentgateway at 0 QPS for 3s and 32 connections...<br>qps: 35501.62qps p50: 0.863ms p90: 1.644ms p99: 1.972ms

DEST,CLIENT,QPS,CONS,DUR,PAYLOAD,SUCCESS,THROUGHPUT,P50,P90,P99<br>litellm,fortio,0,32,3,1114,2982,983.56qps,32.139ms,48.528ms,71.451ms<br>agentgateway,fortio,0,32,3,1114,106525,35501.62qps,0.863ms,1.644ms,1.972ms

==> CPU / memory<br>PAYLOAD CONTAINER SAMPLES AVG_CPU% PEAK_CPU% AVG_MEM PEAK_MEM<br>1024 perf-agentgateway 3 198.90% 482.13% 25.80MiB 34.31MiB<br>1024 perf-litellm 3 68.53% 204.44% 2.15GiB 2.15GiB<br>1024 perf-mock-server 3 18.18% 52.30% 3.05MiB 3.56MiB

==> Checking LiteLLM Rust header<br>x-litellm-rust: true

Full results: github.com/linsun/litellm-agw-perf/results/20260812-215223<br>Visualized results

I asked Cursor to turn the raw benchmark data into charts:

Fixed throughput benchmark

Maximum-throughput tests show the upper limit of each gateway, but they don&rsquo;t provide an apples-to-apples comparison at the same request rate.<br>Since LiteLLM reached approximately 983 QPS in the maximum-throughput test, I ran a second...