Ekka: Automated Diagnosis of Silent Errors in LLM Inference | SyFI Lab

Ekka: Automated Diagnosis of Silent Errors in LLM Inference

Yile Gu, Zhen Zhang, Shaowei Zhu, Xinwei Fu, Jun Wu, Yida Wang, Baris Kasikci

June 29, 2026

Read the paper (arXiv) · ICML 2026 page

TL;DR: LLM serving frameworks ship fast, and sometimes ship silent errors — bugs that degrade output quality without raising any error. Ekka diagnoses them automatically by treating diagnosis as differential debugging: it aligns and compares the intermediate execution states of a buggy framework against a trusted reference, and pinpoints the component where they diverge. On 17 real-world bugs from vLLM and SGLang, Ekka hits 80% pass@1 diagnosis accuracy at ~$30/case , and has already found 4 new bugs confirmed by developers.

Attending ICML 2026? Come find us at Poster Session 5 — Hall A, #2405 , on Wed, Jul 8, 2026 · 5:00–6:45 PM KST . We’d love to chat about AI infra reliability!

Overview

Modern LLM serving frameworks like vLLM and SGLang are remarkable pieces of AI infra engineering: paged attention, radix attention, custom CUDA kernels, continuous batching, and a steady stream of new optimizations. But that velocity has a cost. The more complex and fast-moving the stack, the easier it is to introduce a silent error : a bug where the framework keeps running, returns a perfectly well-formed response, and quietly produces the wrong answer.

Unlike a crash or an assertion failure, a silent error gives you no stack trace and no error message: just output that looks plausible but is subtly, or badly, wrong. Diagnosing these bugs is notoriously hard because of the huge semantic gap between the symptom, say a dropped benchmark score, and the root cause, which might be a misconfigured attention window buried deep in the model stack. Today, tracking one down is a manual, sometimes months-long slog of toggling configs, dumping tensors, and comparing against a reference implementation by hand.

Ekka is an automated diagnosis system that takes on exactly this problem.

A motivating bug

Here is a real silent error from vLLM (#17689). Serving Gemma 3 with vLLM, accuracy on the HellaSwag benchmark dropped by nearly 30% compared to HuggingFace — with no runtime error or warning. Both frameworks were given the same GSM8K prompt:

Prompt

Rory orders 2 subs for $7.50 each, 2 bags of chips for $1.50 each and 2 cookies for $1.00 each for delivery. There’s a 20% delivery fee and a $5.00 tip. What will her order cost?

and produced very different answers:

HuggingFace (reference) → Final Answer: $29 ✓

Subs: 2 × $7.50 = $15. Chips: 2 × $1.50 = $3. Cookies: 2 × $1.00 = $2.<br>Subtotal: 15 + 3 + 2 = 20 . Delivery fee: 20% of 20 = 4. Total: $29

vLLM (buggy) → Final Answer: $100 ✗

Subs: 2 × $7.50 = $15. Chips: 2 × $1.50 = $3. Cookies: 2 × $1.00 = $2.<br>The total cost of the food is 15 + 15 + 15 = 45. Delivery fee: 45 × 0.20 = 9. Total: $100

The generation finishes without error, but the math silently goes off the rails. It took developers three months to trace this to the root cause: a misconfiguration that incorrectly enforced sliding-window attention on all layers instead of 5 out of every 6. From the output alone, there is almost no signal pointing to where in the computation that error originated.

What do silent errors actually look like?

To ground the design, we conducted an empirical study of 90 real-world silent errors collected from vLLM and SGLang GitHub issues. We used the 70 closed issues to understand root causes and diagnosis workflows, and kept the open ones to evaluate Ekka. Three findings shaped Ekka’s design.

Figure 1. Symptom distribution (left) and root-cause distribution (right) of silent errors in vLLM and SGLang.

1. The symptom rarely points to the cause. The most common symptom by far is accuracy regression at 43.8%: superficially valid text that scores worse on benchmarks. The rest are bogus output such as gibberish and repetition loops, plus inconsistent output. A single degraded answer tells you almost nothing about where the computation went wrong.

2. Root causes are spread across the entire stack. Framework implementation at 30.6%, model implementation at 25.5%, and kernel backends at 24.5% all contribute, and about 19.4% of issues are pure numerical precision — floating-point instability with no logical defect at all. Crucially, roughly half of all silent errors originate inside the model stack of model and kernel implementation, so a useful diagnosis tool cannot treat the model as a black box. It has to be model-aware.

3. Developers already do differential debugging — by hand. The most common diagnosis actions are configuration toggling and minimal reproduction, each in over 60% of issues, and comparing against another framework appears in about half of cases, often paired with manually dumping and inspecting activations. This is exactly the workflow Ekka automates.

Figure 2. Diagnosis actions developers perform when resolving...