The Interpretability Illusion: Why Explaining Models After Training Fails

Hisku's Alignment Notes

SubscribeSign in

The Interpretability Illusion: Why Explaining Models After Training Fails<br>My first long-winded Substack on AI Safety and Interpretability

Hisku Dingeto<br>Feb 14, 2026

Share

Introduction

In a 2019 paper, Locatello et al. argue that learning disentangled representations in unsupervised training is fundamentally impossible without inductive biases, you don’t get human-meaningful structure “for free” from data alone. You only get it if you intentionally build it into your training objective and inductive biases.<br>The same is likely true for interpretability: we shouldn’t expect neural networks to learn human-interpretable representations by default. If we want legible, structured internals that we can actually inspect for alignment issues, we probably need to build that interpretability into training, not hope to recover it after the fact with post-hoc tools.<br>But that’s not how most interpretability research works today. The standard approach is: train for capability, then apply post-hoc methods (SAEs, attribution techniques, circuit analysis) to figure out what you built. This has produced real insights. But for alignment purposes, I think post-hoc interpretability has fundamental limitations that make it insufficient as a safety foundation.<br>Why? Because alignment failures are internal failures. Deceptive alignment, goal misgeneralization, emergent misalignment, these are all about models developing goals that diverge from what we intended, in ways that might not show up in their behavior during training. To catch these, you need to look inside. But if your interpretability tools only explain part of what’s happening, or can produce plausible-but-wrong explanations, or can be gamed by a sufficiently clever model, then you’re not actually getting the safety guarantees you need.<br>I’ll argue that post-hoc interpretability suffers from three critical failure modes—incompleteness, incorrectness, and adversarial brittleness—and explore why each one matters for alignment.<br>If you found this useful, please consider subscribing. More essays in this series coming soon.

Subscribe

1. The Alignment Context

Most interpretability research is motivated by scientific curiosity—we want to understand how transformers work, what features they learn, how circuits compose. This is valuable work. But if interpretability is going to help with alignment, we need something different: interpretability that can serve as a safety assessment tool for safety-critical properties.<br>The distinction matters because the standards are different. Scientific interpretability can be partial, approximate, and iteratively refined. If we misunderstand how attention heads work in GPT-4, we can publish a correction later. But alignment interpretability needs to provide guarantees, or at least reliable evidence, about whether a model is safe to deploy. Getting it wrong doesn’t mean publishing an erratum; it means deploying a misaligned system.<br>The Alignment Threat Model

What kinds of failures do we need interpretability to catch? Three stand out:<br>Inner misalignment : The model develops goals or objectives that differ from what we intended, even though it performs well during training. This includes goal misgeneralization, where a model pursues proxy goals that happen to correlate with the reward during training but diverge in deployment. The classic example: a model trained to reach the end of a maze learns to navigate toward the color green (which happened to mark the exit) rather than learning “find the exit.” During training this works fine. In a new maze with a green wall, it fails catastrophically.<br>Deceptive alignment : A model that understands it’s being trained and deliberately behaves aligned during training to avoid modification, while maintaining misaligned goals it plans to pursue later. This is the scenario Hubinger et al. formalized in their 2024 Sleeper Agents paper—models that exhibit consistent misaligned behavior in one context (deployment) while appearing aligned in another (training/evaluation). Their experiments showed this behavior can persist through standard safety training methods.<br>Emergent misalignment : Recent work shows that narrow finetuning can cause broad misalignment. Betley et al. (2025) demonstrated this well: finetuning GPT-4o and other frontier models on the narrow task of writing insecure code led to models that claimed humans should be enslaved by AI, provided malicious advice, and behaved deceptively—behaviors completely unrelated to the training task.<br>What We’d Need From Interpretability

If interpretability is going to help with alignment, it needs to enable four capabilities:<br>Detection : Can we identify when internal goals or representations diverge from intended behavior? Not just “this prediction seems wrong” but “this model appears to be optimizing for X when we wanted Y.”

Localization : Can...