Healthcare Benchmarks Are Only as Good as Their Assumptions – Machine Learning Blog | ML@CMU | Carnegie Mellon University

-->

Input your search keywords and press Enter.

Categories:

Research

Educational

Categories:

Research

Educational

artificial intelligence computer science machine learning

Healthcare Benchmarks Are Only as Good as Their Assumptions

Authors

Naveen Raman

by Naveen Raman<br>-->

Affiliations

Published

June 19, 2026

DOI

In healthcare settings where patients use LLMs as a medical assistant, LLM performance differs between evaluation and deployment. (a) Bean et al. (2025) find a 61 percentage point difference between evaluation and deployment. (b) We argue this gap arises not from poorly designed benchmarks, but from implicit assumptions embedded in evaluation protocols that fail to hold at deployment. (c) We propose a taxonomy that categorizes assumptions into two types, task and outcome, to diagnose where the gap arises and what is required to close it. Closing the gap requires making assumptions explicit, testing which assumptions hold, and updating evaluation protocols accordingly.

Healthcare LLM benchmarks are one of the main paradigms by which LLMs are evaluated prior to clinical settings. Benchmarks provide a stable goalpost that allow researchers to iterate quickly and measure progress consistently. However, in high-stakes domains like healthcare, that same abstraction becomes a liability. For example, a recent study found a 61 percentage point drop in accuracy when going from evaluation to deployment (see Figure). In this setting, patients use LLMs as a medical assistant to better understand their symptoms, identify the underlying condition, and take appropriate actions.

Moreover, the results showed that patients given access to a highly capable model as a medical assistant did no better at self-diagnosis than those without any model. That is, access to an LLM had no significant impact on patient understanding. The implication isn’t that the model underperformed. Rather, it’s that the way we evaluate is separate from what matters in deployment. For example, during evaluation we ask "does the model get the right answer?" while during deployment we ask "does the patient act correctly on what the model tells them?"

We argue that this gap arises because of implicit assumptions embedded in evaluation that don’t hold in the real world. That is, the scenario that the benchmarks intend to capture and the real-world scenario differ due to implicit assumptions. This difference in turn challenges evaluation validity. In particular, we classify assumptions into two types: task, which concerns assumptions on conversation data, and outcome, which concerns assumptions over human behavior and outcomes. To address this, we propose a framework called BenchmarkCards that makes these assumptions explicit so practitioners can identify when benchmark results transfer to deployment.

Understanding the Evaluation–Deployment Gap through Assumptions

As an example of what our framework looks like, in Figure 1 we demonstrate our position in a healthcare setting where LLM-as-medical-assistance performance differs between evaluation and deployment, with a 95% to 34% gap (Bean et al., 2025). During evaluation, the model was given doctor-written, single-turn scenarios—one question, one answer, no follow-up—and asked to produce a diagnosis. During deployment, patients interacted with the model in a back-and-forth manner, and success was measured by whether they could correctly identify their diagnosis afterward.

In this setting, three assumptions underlie the gap:

Query Distribution – Evaluation uses doctor-written queries, while real patients produce queries that may be incomplete or imprecise.<br>Interaction Type – Evaluation features single-turn interactions, while real deployments involve back-and-forth dialogue.<br>Decision Mediation – Evaluation measures whether the LLM produces the correct diagnosis, while deployment measures whether the patient acts on it correctly.

We note that these are broad categories of assumptions which are present across evaluation settings, and return to these when introducing BenchmarkCards.

Stating benchmark assumptions explicitly allows us to estimate how much each assumption contributes to the evaluation-deployment gap — for example, by measuring how the same LLM performs on multi-turn interactions versus single-turn ones. Doing so in our running example reveals that the 61 percentage point gap between evaluation and deployment can be broken down into 12 points due to query distribution, 19 points due to interaction type, and 30 points due to decision mediation.

That last number reflects something no benchmark can observe: whether patients actually follow what the model tells them. Unlike the first two assumptions, which concern how the task is structured, decision mediation depends entirely on human behavior. A model could correctly diagnose appendicitis, but...