Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer | OpenReviewGo to ICLR 2026 Conference homepage
Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer
Simon Schrodi, Elias Kempf, Fazl Barez, Thomas Brox
Published: 26 Jan 2026, Last Modified: 11 Apr 2026ICLR 2026 Postereveryonesince 08 Oct 2025">EveryoneRevisionsBibTeXCC BY 4.0
Keywords: subliminal learning, hidden bias transfer, LLMs, finetuning, distillation, alignment, safety<br>TL;DR: We studied how language models transfer hidden biases through unrelated data (subliminal learning) and, using controlled experiments, identified a small set of context-sensitive divergence tokens that primarily drive subliminal learning.<br>Abstract: Language models can transfer hidden biases during distillation. For example, a teacher that "likes owls" can make its student "like owls" too, even when the training data consists only of lists of numbers. This surprising phenomenon is called *subliminal learning*. Subliminal learning can be expected under soft distillation, where the student is trained on the teacher's full next-token distribution. But the fact that this also occurs under hard distillation—where the student only sees sampled tokens—raises a deeper question: *when and how does subliminal learning actually occur?* We answer this question through controlled experiments and mechanistic analysis. Our results show that subliminal learning does not need (global) token entanglement or logit leakage. Instead, it comes down to a small set of *divergence tokens*—rare cases where teachers with different biases would predict different tokens. Masking out these tokens mostly removes the hidden bias transfer. Mechanistically, divergence tokens reveal that early layers are critical. Surprisingly, finetuning even a single such early layer is sufficient for subliminal learning. Finally, we find that subliminal learning is fragile. Even small changes, like prompt paraphrasings, are usually sufficient to suppress it.<br>Primary Area: alignment, fairness, safety, privacy, and societal considerations<br>Submission Number: 4323
Loading
OpenReview is a long-term project to advance science through improved peer review with legal nonprofit status. We gratefully acknowledge the support of the OpenReview Sponsors. © 2026 OpenReview