ResidencyRL: Can AI Learn Clinical Judgment in Simulation? | echohive

Skip to content

Membership →

01 / THE PROBLEM

Knowing medicine is not the same as conducting a consultation.

A medical exam gives the model a question and the facts needed to answer it. A real consultation does not. The clinician must decide what to ask next, notice what is missing, revise a diagnosis, choose a safe action, explain it, and know when to escalate.

That makes clinical reasoning a sequence problem . One early assumption can change every later question. One missed red flag can make an otherwise fluent answer unsafe.

ResidencyRL, submitted on August 7, tests whether reinforcement learning across complete simulated encounters can improve this process. The starting point was Gemini 3.5 Flash. The training environment made it practice, receive structured feedback, and try again.

STATIC TESTfacts → answer The relevant information is already present.

CLINICAL PROCESSask → update → act → verify The next useful fact must be discovered.

02 / THE SIMULATED RESIDENCY<br>Practice happens inside a generated clinical world.

The model is rewarded for the quality of the whole encounter, not only its final diagnosis.

01Generate a case<br>A scenario defines the condition, history, required questions, red flags, management, and safety constraints.

02Simulate the patient<br>An LLM patient reveals information through dialogue and can use complex or adversarial behavior.

03Run the encounter<br>The policy model asks questions, updates its view, uses tools, proposes management, and writes documentation.

04Score the trajectory<br>Structured judges evaluate diagnosis, management, intake, communication, documentation, style, and safety.

05Reinforce better behavior<br>The training update makes stronger complete trajectories more likely in the next round.

TRAINING MIXMost cases taught routine breadth. Smaller sets targeted process and danger.

49,870 general telehealth<br>81 conditions<br>5,000 history<br>taking<br>2,583 adversarial<br>safety

GET AMPLIFIEDStudy the systems behind the capability.<br>Follow practical work on agent environments, evaluation, reinforcement loops, and the human judgment that makes them useful.

Explore Get Amplified →

03 / WHAT IMPROVED<br>The strongest gain was a better search for missing information.

Held-out adversarial encounters tested whether the model would close too early on a diagnosis or overlook a safety signal.

Before and after ResidencyRL training<br>Gemini 3.5 Flash ResidencyRL

High diagnostic score Higher is better

81.0% 88.0%

+7.0 pts

Missed critical question Lower is better

65.5% 43.5%

−22.0 pts

Missed red flag Lower is better

45.5% 31.5%

−14.0 pts

Under-triage Lower is better

4.0% 4.5%

No gain

The gain and the residue must be read together. A 31% relative reduction in missed red flags is meaningful. A 31.5% remaining miss rate in these difficult simulations is also too high for an autonomy claim.

MULTIVISIT TRANSFER+8.34 Management reasoning<br>On 120 AMIE Mx scenarios, the score rose from 80.07% to 88.41%. Communication rose by 8.50 points and clinical skills by 5.26.

SPECIALIST TRANSFER42.9% Oncology comparison wins<br>Across 300 unseen oncology cases, ResidencyRL won 42.9% of composite comparisons versus 18.6% for the baseline. The rest were ties.

EXTERNAL BENCHMARKS↗ Promising, not conclusive<br>AgentClinic and CRAFT-MD results generally moved in the favorable direction, but most individual gains were not statistically significant.

04 / WHAT THE EXPERT PREFERENCE MEANS<br>Clinicians saw a better consultation. They did not validate an AI replacement.

87.6% overall preference

97 valid blinded side-by-side comparisons. ResidencyRL versus its baseline model.

90.7%preferred for completeness of information gathering

75.3%preferred for management-plan appropriateness

66.0%preferred for diagnostic assessment

96.9%preferred or tied for management-plan safety

PLAIN LANGUAGE<br>The trained model was usually more complete and useful than the same model before this training. The study did not compare its full performance with physicians, did not deploy it in routine care, and did not measure whether patients became healthier.

05 / THE THREE-MONTH FRONTIER<br>Medical AI is assembling a larger capability stack.

Only research published from May 12 through August 12, 2026 is used here.

Aug 7Practice complete consultations ResidencyRL

Jun 17Remember across three visits AMIE disease management

May 14Use images, ECGs, and documents Multimodal AMIE

Aug 10See and speak in real time AMIE Video

Jun 17Act inside a simulated EHR MIRA

May 14Simulate possible patient responses SepsisAgent

Video closes one gap and reveals others.<br>An August 10 AMIE Video study combined low-latency dialogue, clinical planning, and audio-visual perception. Actor-based evaluation was strong, while fine anatomical precision, subtle affect, and fast movement remained difficult.

Longitudinal care needs memory and evidence.<br>A June Nature study tested management...