Are my evals lying to me? - Hop Labs

Hop Labs

SubscribeSign in

Are my evals lying to me?<br>…and when they do, how will I know?

Hop Labs<br>Jul 15, 2026

Share

Photo by Google DeepMind on Unsplash<br>In 2021, researchers at Michigan Medicine evaluated the Epic Sepsis Model against their own patient data and published a sobering result. Epic’s predictor was already live in hundreds of American hospitals, deployed to catch the single deadliest consequence of infection in clinical care. On Epic’s own internal evaluations, the model looked strong; but against real patients in Michigan, its performance plummeted to little better than a coin flip. It triggered alerts on 18% of all admitted patients, yet still missed two-thirds of those who actually had sepsis1.<br>A STAT News investigation2 identified a likely culprit among the model’s inputs: whether a clinician had already ordered antibiotics — an action typically taken only when sepsis is already suspected. Consequently, a feature meant to predict a diagnosis was actually relying on a diagnosis a human had already made. “Antibiotics ordered” was supposed to be just one clue among many. Instead, it leaked a conclusion the model was supposed to reach independently based on raw patient data.<br>Thanks for reading! Subscribe for free to receive new posts and support my work.

Subscribe

On the validation data, the antibiotic order and the final diagnosis traveled together, creating an illusion of accuracy. But while the validation score successfully measured this correlation, it couldn’t predict whether it would hold up in the realities of live deployment.<br>It’s a near-textbook evaluation error, specifically a form of target leakage: the model trained on information about the outcome that wouldn’t actually be available — or mean the same thing — at the true moment of prediction.<br>For any model tackling any problem, the hardest part of evaluation is keeping the metric aligned with reality. Every team building machine learning systems eventually runs into this same wall. We treat evaluations as unimpeachable — the fixed point against which everything else is measured. In reality, an evaluation is just a measurement tool, and tools can be miscalibrated, biased, or simply pointed at the wrong thing.<br>A bad model is relatively straightforward (in the scheme of AI problems). A broken evaluation, unfortunately, tends to flatter you. It tells you exactly what you want to hear, and without an independent signal to contradict it, you won’t discover the flaw until the system meets reality and something goes terribly wrong.<br>The urgent question is less: “Are my evals lying to me?” (They are, or they soon will be.) Instead, it’s: “When my evals do lie, how will I know — and what can I do about it?”<br>An eval is a proxy for what you care about

When used well, AI/ML evaluations serve a critical function. They expose regressions, guide iteration, compare systems, reveal trade-offs, and reduce risk. Used poorly, they create false confidence, reward gaming, hide edge-case failures, and collapse complex system behavior into a single, misleading number.<br>You can run your model against a test set and read off an accuracy score, but that number is rarely what you actually care about. What you care about is how the model behaves in production — against real users, over months, and on inputs you never anticipated. The test set is worth exactly as much as it resembles the live outcomes, and no more. This is the oldest problem in applied AI, but it still catches teams and researchers off guard.<br>Large language models add a new layer of difficulty. With a predictive model, you can at least define the boundaries of the input space: a fraud model sees transactions; a sepsis model sees vitals and clinician orders. But the space of things a user might type into a prompt box is unbounded and constantly shifting. You cannot characterize it the way you would a predictive model’s inputs — in fact, it is rarely clear what “characterizing” it would even mean.<br>Your team can build an evaluation dataset from everything it imagines users will ask. But when those users actually arrive, they write in fragments, misspell words, and ask questions nobody pictured — or questions nobody should have asked — in proportions nobody could have guessed.<br>In the sepsis model, the evaluation and reality diverged because a single factor didn’t actually mean what it did in testing. But with language models, they come apart before you’ve measured anything at all: your test set is drawn from the inputs you could imagine, while reality is also drawn from the inputs you couldn’t. A high score tells you only that the model handled the questions you thought to ask; it says nothing about the ones users actually brought. The proxy and the target start miles apart, and the evaluation can’t show you the distance between them.<br>When using the model breaks it

There’s another, possibly more stubborn reason that evaluations and reality part ways: the...