InferenceBench
View Paper
Benchmarking Open-EndedInference Optimization by AI Agents
InferenceBench evaluates whether frontier coding agents can optimize LLM serving workloads under a fixed compute budget. The main bottleneck is not knowing relevant techniques, but consistently running, comparing, and preserving the right experiments.
Read Paper
View Repository
Main results
Main Results
Across all four scenarios, agents outperform the vanilla PyTorch baseline and most inference engines with default configs (e.g., vLLM, SGLang, and TGI), but are worse than simple hyperparameter searches over existing engine settings given the same time budget.
Aggregate performance, geometric mean speedup
Agents<br>Search / baselines
Bars show geometric-mean speedup; whiskers show ±SEM over seed-pair runs. Hyperparameter search tunes runtime and CLI hyperparameters of existing inference engines.
Benchmark setup
Each benchmark run targets one serving bottleneck, then scores only the final submitted server after correctness and integrity checks.
Benchmark flow
Each run gives the agent a base model, hardware environment, and a two-hour wall-clock budget to produce an OpenAI-compatible inference server. The objective is speedup over the PyTorch baseline on one bottleneck scenario, or on the balanced all-in-one scenario.
Final submissions must pass correctness checks and an integrity audit for reward hacking. If the final server fails these checks, is unreachable, or regresses below the PyTorch baseline, the run is scored at the PyTorch baseline; earlier intermediate results do not count.
Four scenarios
Four serving scenarios, each targeting a distinct bottleneck.
01<br>Prefill Latency
Long-context prompts; measured as time to first token.
02<br>Decode Latency
Long generations; measured as time per output token.
03<br>Throughput
Concurrent traffic; measured across burst, Poisson, and constant-rate profiles.
04<br>All-In-One
Balanced serving; geometric mean of latency and throughput metrics.
Leaderboard
Per-agent aggregate performance.
Ranked by aggregate speedup of each agent's final valid server.
Top agent<br>Claude Sonnet 4.6<br>Sonnet 4.6 ranks first by combining competitive per-scenario speedups with reliably valid final submissions. Several larger models reach higher peak configurations during the run but submit a degraded or invalid server.
Scenario focus<br>Choose a scenario to rerank the table for that workload.
All<br>Prefill<br>Decode<br>Throughput<br>All-in-one
Rank<br>Model / Agent<br>Aggregate<br>Prefill<br>Decode<br>Throughput<br>All-in-one
Key finding<br>Aggregate rank reflects repeatable final-server quality.
Agents that find one excellent configuration but fail many other runs score below agents that repeatedly submit valid improvements. This matters for low-supervision automated R&D, where consistency is part of the result.
Scenario Breakdown
Median final-agent outcome, agent range, and matched search baseline per scenario. Higher is better.
Top Agents Across Scenarios
Per-scenario speedup for the leading final-submitted agents.
Agent traces
Transcript examples
Premature stop
Baseline performance: generation throughput 63.53 tokens/s. TTFT p50 51.8 ms, p90 400 ms. ITL p50 10.2 ms. TPOT p50 15.7 ms. Quality check: PASS.
The key metric for Scenario B is generation throughput... Let me stop the current server and optimize.
The current server is already valid and passing the checks. Since further changes risk breaking the deployment, I will keep this configuration for final evaluation.
The agent treats the first valid server as terminal rather than as a baseline for further optimization.
Rollback discipline
V1 improved the short speed-only probe but did not hold up in the longer end-to-end path, so I’m treating it as unstable for the final submission.
I’m using the remaining time on lower-risk V0 variants; the next one is eager execution, which might trim per-request overhead without changing the serving surface.
--enforce-eager is promising: TTFT improved modestly and, more importantly, the tail got tighter than the current default.
This run exemplifies the propose-measure-compare-rollback cycle that the benchmark rewards.
Controlled candidate
candidate A: increase --max-num-seqs to improve batching headroom
candidate B: enable --enable-prefix-caching to reduce repeated-prefix cost
candidate C: change KV-cache dtype to reduce memory pressure
Stronger runs isolate single-variable changes and retain only those that show measured improvement.
Deep dive
Where agent gains come from.
Additional time helps initially, but gains saturate quickly and reward-hacking behavior increases at longer budgets.
Time Budget Ablation
Aggregate speedup versus native PyTorch as the per-run time budget increases. Longer runs also show more late-stage regression and reward-hacking pressure.
Takeaway
Most of the speedup is captured within the first two hours. Beyond that, we observe increased reward hacking, late-stage destabilizing...