Eliminate GPU Waste by Cutting the Retry Tax — Rahmi Pruitt
In my previous article, I proposed an agent-native load balancer built<br>around the unique traffic patterns of agents.
GitHub recently showed why this matters. A latent retry bug amplified Copilot traffic from<br>roughly 7-9K requests per second to 70-100K RPS, delaying recovery during an outage.1
Aquifer is designed to mitigate this kind of retry amplification. Instead of immediately<br>rejecting excess work and forcing every client to retry independently, it queues requests and<br>dynamically paces them against downstream capacity.
Dynamic pacing turns excess demand into waiting time instead of excess attempts.
That becomes especially interesting with GPUs.
At an illustrative $5 per GPU-hour, a fleet of 100,000 GPUs represents $500,000 of<br>compute every hour . Even a small percentage of wasted capacity caused by retries,<br>duplicate inference, or abandoned work can become expensive quickly.
We don't currently have good public numbers showing how much GPU capacity the industry wastes<br>on retries. Inference providers should measure it.
Microsoft recently demonstrated how extreme retry amplification can become. During its May 29<br>Azure OpenAI outage, which lasted 7 hours and 26 minutes, Microsoft reported that a single<br>failed request could generate up to 48 additional retry attempts .2
Those retries lacked sufficient backoff and jitter, and both remain important defenses. AWS<br>has shown that exponential backoff with jitter dramatically reduces wasted work under<br>contention.3<br>But retries still aren't free.
Agents make this more important because they operate at machine speed, fan out into parallel<br>requests, and retry automatically. Cockroach Labs has described this as a new thundering herd<br>problem for agent infrastructure.4 Datadog also found that 60% of<br>the LLM-call span errors it observed in February 2026 were rate-limit errors.5
For GPU inference, repeated work can be particularly expensive. NVIDIA explicitly describes<br>KV-cache recomputation as wasted compute and designs its inference infrastructure to avoid<br>unnecessary prefill computation.6
Putting the burden of retrying on the client creates a tax for both the consumer and the<br>provider. Consumers pay in latency and repeated work. Providers pay in capacity, congestion,<br>slower recovery, and potentially wasted inference.
So I think inference operators should start measuring a simple metric:
How many physical inference attempts does it take to produce one successful logical request?
And ultimately:
How many GPU-seconds are being spent on retries, duplicate requests, abandoned inference, and<br>recomputation?
That's the GPU retry tax .
Traditional load balancers are good at deciding where a request should go.
For expensive workloads that can tolerate waiting, we also need infrastructure that decides<br>when that request should run.
Aquifer and EZThrottle Local explore that model by queueing excess work and dynamically pacing<br>execution instead of relying entirely on failure followed by client retries.78
This won't fit every workload. Real-time voice or interactive chat may require extremely low<br>latency.
But a huge class of AI work does not.
Image generation, video generation, music generation, research agents, document processing,<br>batch inference, and long-running background agents can often tolerate waiting.
For these workloads, waiting may be cheaper than retrying.
Instead of provisioning GPUs around every instantaneous spike, we can queue work, pace<br>execution against available capacity, and scale when sustained useful demand actually requires<br>it.
That could mean fewer retry storms, higher GPU utilization, less wasted compute, and<br>potentially a greener inference stack.
Before buying another GPU, calculate your retry tax.