DeepSeek V4 Flash performance on a single MI300X
DeepSeek V4 Flash performance on a single MI300X
What a single GPU actually delivers when the clients are<br>autonomous coding agents doing real work, not synthetic benchmark load.<br>Built on the open-source<br>deepseek-v4-flash-mi300x<br>serving stack.
Most published LLM serving numbers come from synthetic load: fixed prompt<br>lengths, ShareGPT replays, evenly spaced arrivals. I wanted to know what a<br>single MI300X delivers when the clients are coding agents working on real<br>repositories, because that workload looks nothing like the benchmarks. Every<br>number here comes from a 30-minute measured window tied to a recorded<br>serving-config fingerprint.
The short version: one MI300X comfortably serves 32 concurrent coding<br>agents on DeepSeek V4 Flash. Past 32 agents this configuration does not<br>degrade gracefully, so I treat 32 as the practical ceiling.
Generation, 32 agents<br>582<br>tokens/sec
Prompt processing<br>78.3K<br>tokens/sec
Avg prompt per request<br>53,194<br>tokens in, 390 out
Prefix cache hit<br>93%<br>the load-bearing metric
The workload
The load generator is a bench harness I built for this: N lanes, each lane<br>an opencode agent session working on a clone<br>of a real open-source repo (req, oban, supabase/realtime, livebook, flask,<br>pydantic, hono, excalidraw). Each lane picks up a generated task, works it<br>like a contributor would (reads code, edits, runs the test suite, commits),<br>and either gets its change merged or times out. All inference goes through<br>one vLLM instance. Everything is scraped into Postgres every 5 seconds.
What makes this workload different from synthetic load, measured during<br>the 32-lane window:
The average request carries 53,194 prompt tokens and<br>generates just 390 . Agents resend their whole growing<br>conversation every turn.
That only works because of prefix caching: the hit rate under steady load<br>is 93 to 99% . When it drops, everything drops.
Arrivals are bursty and phase-correlated. Agents block on their own test<br>suites, then all come back at once.
The tasks are deliberately hard and most hit their timeout: a typical<br>30-minute window at 32 lanes completes ~2,600 agent turns, merges 3 or 4<br>tasks, and abandons ~25 at the timeout. This is a serving benchmark, not a<br>coding-ability one; the tasks exist to generate honest load.
Hardware and stack
One AMD Developer Cloud box: 1× MI300X (192 GB HBM3), 20 vCPU,<br>236 GB RAM. The agents and their test suites run on the same box as the<br>engine, which is realistic and occasionally consequential.
Serving is the pinned open-source<br>deepseek-v4-flash-mi300x<br>stack: vLLM nightly 0.26.1rc1.dev229 (ROCm), DeepSeek V4 Flash<br>at 262k context, MLA + DSA sparse attention, speculative decoding (~49%<br>draft acceptance, stable across every window), and a 96 GB CPU KV offload<br>tier. Plus two sets of overlays from this repo: two upstream vLLM bugfixes<br>for a KV-zeroing kernel that was page-faulting the GPU under load, and the<br>AITER GEMM tuning described below.
Concurrency sweep
30-minute windows per lane count, engine warm, cache hit gated above 90%<br>before each window starts. The first three configurations are the untuned<br>stack; the fourth is the same 32-lane load after tuning, on a freshly<br>provisioned box.
Generation throughput tokens/sec by agent count
200<br>400<br>600<br>8 agents: 358 tok/s<br>16 agents: 448 tok/s<br>32 agents: 533 tok/s<br>32 agents, tuned: 582 tok/s<br>358<br>448<br>533<br>582
16<br>32<br>32 tuned
Prompt throughput tokens/sec by agent count
20K<br>40K<br>60K<br>80K<br>8 agents: 54,020 tok/s<br>16 agents: 75,517 tok/s<br>32 agents: 64,441 tok/s<br>32 agents, tuned: 78,262 tok/s<br>54.0K<br>75.5K<br>64.4K<br>78.3K
16<br>32<br>32 tuned
agentsgen tok/sgen tok/s per agentprompt tok/sTTFT p50TTFT p99cache hit
835844.854,020≤0.75s≤5s99%<br>1644828.075,517≤1s≤5s98%<br>3253316.764,441≤2.5s≤20s94%<br>32, tuned58218.278,262≤1s≤5s93%
The GPU runs at 97% utilization drawing ~717 W during the 32-lane window,<br>and the queue stays near empty: the engine is keeping up with demand, not<br>saturating. Scaling from 8 to 32 lanes costs about half a second of median<br>TTFT and buys 63% more generation throughput.
What it costs
The box rents for $2/hour. Against the tuned 32-agent window, that works<br>out to:
Per 1M generated tokens<br>$0.95<br>582 tok/s sustained
Per 1M prompt tokens delivered<br>$0.007<br>97.9% served from cache
Per 1M prompt tokens computed<br>$0.33<br>the uncached 2.1%
The delivered prompt number is only that low because caching absorbs<br>nearly everything. Of the 142M prompt tokens in the window, 93.0%<br>came from the GPU prefix cache, 4.9% from the CPU KV offload<br>tier, and only 2.1% were actually computed. Keep that in mind<br>when comparing against per-token API pricing, where cached input is billed<br>too; the computed-token price is the number that reflects work the GPU<br>actually did.
What tuning bought
The stack ships AITER GEMM tuning for prefill sizes this workload never<br>produces (the scheduler caps prefill chunks below every tuned shape), so<br>~70% of engine log lines were...