What Is LLM-as-a-Judge, Really? A Field Guide to the State of the Art in 2026 - Karthika Raghavan
You are using an outdated browser. Please upgrade your browser to improve your experience.
What Is LLM-as-a-Judge, Really? A Field Guide to the State of the Art in 2026
I wanted to actually understand LLM-as-a-judge instead of nodding along the way I used to with CAP theorem before I’d actually read the paper. So I built a proper research pipeline, tried to find something genuinely novel to say about it, and watched all five of my “novel” ideas get killed by papers published in the last few months. Here’s everything I learned along the way — with the receipts.
Karthika Raghavan
Software Engineer • Distributed Systems • LLM Infrastructure
Follow
Vancouver, BC
GitHub
Custom Social Profile Link
-->
1. What Is LLM-as-a-Judge?<br>2. Why It Had to Exist<br>3. The Papers That Built the Field<br>4. Where It’s Actually Used Today<br>5. The Bias Zoo<br>6. Where Judges Fall Off a Cliff<br>7. When Judges Get Attacked on Purpose<br>8. The Mitigation Toolkit, Ranked by How Solved It Actually Is<br>9. The Honest Interlude: I Tried to Find You Something Novel<br>10. Where the Frontier Actually Is<br>11. The War Room Reference Table<br>Conclusion: The Fundamentals Are Settled. The Frontier Is Not.<br>References
Let me be honest about how this post happened, because it’s a better story than the post itself.
I wanted to learn LLM-as-a-judge properly — not “yeah, you use an LLM to grade another LLM’s output” properly, but actually understand where it came from, where it breaks, and what people smarter than me are doing about it. And since I’d just spent a week writing a reactive hot-take about someone else’s product launch and gotten (correctly) called out for having zero citations and zero original insight, I decided to do this one right: real research, real sources, and if I was going to claim something novel, it had to survive someone actively trying to kill it.
So I spun up a small army of research agents. Six of them went and surveyed the field — foundations, biases, calibration, mitigations, the 2025-2026 frontier, and how it’s actually used in production. Then a synthesis pass consolidated all of that into five candidate “gaps” in the current toolkit. Then I had agents develop each gap into a concrete, technically-grounded proposal. Then — and this is the part that actually taught me something — I had three independent skeptics per proposal try to refute each one: find the prior art, poke the mechanism, default to “this doesn’t survive” unless proven otherwise.
All five proposals died. Every single one. Not because the ideas were bad — the mechanisms were sound — but because someone had already published almost exactly the same thing, in some cases within the last twelve weeks. I’ll walk through the wreckage later in this post, because it turned out to be the most useful part of the whole exercise. But first, the actual field guide, because that’s what I came here to learn.
1. What Is LLM-as-a-Judge?
Picture a professor with 50,000 essays to grade by Friday. They can’t read all of them — nobody can. So they hire a teaching assistant: someone smart, fast, and (crucially) consistent, who reads a rubric once and then applies it identically to essay #1 and essay #49,999. The TA isn’t the professor. The TA can be wrong, tired, biased toward students who write in a font the TA likes, or unable to tell a genuinely brilliant unconventional answer from a well-formatted one that says nothing. But the TA is fast enough that grading actually finishes, and reliable enough — most of the time — that the grades mean something.
LLM-as-a-judge is that TA, and the “essays” are LLM outputs: chatbot responses, RLHF preference pairs, agent trajectories, summaries, code. Instead of a human rater or a fixed answer key, you prompt an LLM to score or compare outputs — pointwise (“rate this 1-10”), pairwise (“which of these two is better”), or via a rubric (“does this satisfy criteria A, B, and C”). The judge doesn’t need to be a different model than the one being graded, though — as we’ll get to — that turns out to matter a lot.
The judge sits between candidate outputs and a verdict. The human rater doesn't disappear — it moves to a gold-set spot check.
2. Why It Had to Exist
Two gaps forced this into existence, and both are obvious in hindsight.
Gap one: the old metrics don’t work for open-ended text. BLEU and ROUGE — the classic n-gram overlap metrics — were built for machine translation, where there’s a roughly “correct” answer to compare against. They fall apart on summarization, dialogue, or anything with legitimate diversity of good answers. G-Eval (Liu et al., Microsoft, EMNLP 2023) showed that GPT-4, prompted with chain-of-thought reasoning and a structured “form-filling” scoring template, correlated with human judgment far better than any n-gram metric — 0.514 Spearman correlation on summarization, a genuinely large jump.
Gap two: human...