Testing LLM Concurrency on Consumer Hardware — ai.2it.onl
☰ Site nav
ai.2it.onl<br>personal blog · agentic computing experiments
Test rig
CPURyzen 9 9950X3D · CCD0
RAM32GB DDR5 6400 1:1
GPURTX 5060 8GB GDDR7
Display3 monitors under load
Testing LLM Concurrency on Consumer Hardware
2026-08-02<br>·<br>~14 min read<br>·<br>benchmark<br>15 models<br>360 runs
I recently watched a YouTube video of someone testing a server-grade LLM hardware<br>setup, pushing it to see just how much concurrency it could actually handle.<br>It got me thinking: what can your own — perhaps a bit above-average —<br>“gaming” / “workstation” PC really do? Especially within the limits of my<br>RTX 5060 and its fast but limited 8GB of VRAM. I’ve been thinking about building a game<br>or simulation driven by a high agent count, and I wanted to know what the feasible limit<br>really is. That question brought me to these tests — and the results you’ll find<br>below: the rig, the method, and every number and chart for all 15 models,<br>individually and side by side. A note on authorship: that opening paragraph is my own<br>words — the rest of this post is mostly AI-generated, though I have carefully gone<br>over all of it, and every number has been checked against the raw benchmark data before<br>publishing. And the dataset isn’t done yet — I plan to keep testing more models<br>and will add them to the list as I do, so keep a lookout for new entries in the sidebar.
TL;DR
Concurrency scales on consumer hardware — pooled agents multiplied throughput<br>8.7x at best. The top all-in result was<br>MiniCPM5 1B at 983 tok/s<br>(433% of its solo speed).<br>The worst was a flatline: Qwen3.5 0.8B with MTP enabled never scaled at all<br>(~180 tok/s at any concurrency) because multi-token prediction serializes requests —<br>with MTP off the same model hit 787 tok/s.<br>Thinking models hide a huge share of their budget in reasoning_content (up to 100%),<br>and time-to-first-token degrades from milliseconds to tens of seconds under load. Errors across all<br>360 runs: zero .
Test rig
One desktop, fully loaded while every run executed. This is a dirty daily-driver environment — not a clean lab bench — and that was the point: the results are what real consumer hardware actually does.
CPU · compute<br>Ryzen 9 9950X3D
Zen 5 V-Cache flagship. Core affinity is hard-locked to CCD0 so every LLM loop stays on the 3D V-Cache cores — zero cross-CCD thread-hopping on the Infinity Fabric.
RAM · bandwidth<br>32GB DDR5 6400
UCLK locked 1:1 with the memory controller for the lowest possible system↔GPU transfer latency during CPU-side MoE offload and high-concurrency prefill pooling.
GPU · acceleration<br>RTX 5060 8GB GDDR7
Core + VRAM overclocked to squeeze maximum raw bandwidth out of the 128-bit bus. Every model was fully offloaded — the GPU is the engine room.
Environment · dirty load<br>3 monitors, always on
2x ultrawide + 1x 4K primary, with a hardware-accelerated multi-tab Chrome profile running through every single benchmark run.
LM Studio configuration
Identical server settings for every model — differences (MTP, KV cache quant, context caps) are called out per model on its page.
⚒ Decode & scheduling
GPU offloadMax for model
CPU threads8 · CCD0
Eval batch size2048
Physical batch size512
Max concurrency24
Flash attentionOn
● Memory & context
Context length34304 · max
Unified KV cacheOn
Offload KV to GPUOn
Keep model in memoryOn
Try mmap()On
KV cache quantF16 · Q8_0 · Q4_0
⚡ Decoding strategy
Speculative decodingOff
MTP exceptionQwen3.5 0.8B
KV quant rulesF16 ≤1.5B · Q8_0 2B · Q4_0 3B+
Context cap32k or 8k · per model
Methodology
The harness is a zero-dependency Node agent loop (benchmark/benchmark.js) plus a<br>sweep driver (benchmark/sweep.js) streaming against LM Studio’s local OpenAI-compatible<br>endpoint. Each model got 24 sequential runs —<br>one per concurrency level 1–24 — of 60 seconds wall time . Each agent holds a<br>12,000-token budget , runs a multi-turn loop until budget or timeout, and samples<br>temperature 0.9.
Token-level timing captures TTFT (time to first token), per-stream token gaps,<br>content vs reasoning tokens, and per-agent/combined throughput. Reasoning is counted differently<br>per model family: reasoning_content streams (MiniCPM5, Qwen-DeepSeek, Chronos, Qwen3.5),<br>inline ... tags stripped from content (Gemma Heretic), or not at all<br>(the clean non-thinking models). Per-model pages note which.
One honesty flag: some per-agent numbers carry * — agents that sat queued for<br>~11–16s and then generated in a short burst report inflated per-agent tok/s. Those are<br>timing artifacts; combined throughput and TTFT are the trusted columns .
Leaderboard — side by side
983<br>best combined tok/s (MiniCPM5 1B)
872%<br>best scaling (Unsloth Ministral 3 3B (2512))
7/15<br>thinking models
errors in 360 runs
best in column<br>worst in column<br>podium (top 3)<br>lower is better...