Small Models Can Introspect, Too
Small Models Can Introspect, Too
Posted<br>December 12, 2025
Recent work by Anthropic showed that Claude models, primarily Opus 4 and Opus 4.1, are able to introspect--detecting when external concepts have been injected into their activations. But not all of us have Opus at home! By looking at the logits, we show that a 32B open-source model that at first appears unable to introspect actually is subtly introspecting. We then show that better prompting can significantly improve introspection performance, and throw the logit lens and emergent misalignment into the mix, showing that the model can introspect when temporarily swapped for a finetune and that the final layers of the model seem to suppress reports of introspection.
Enjoy! This was written as part of the Thebes Funemployment Arc, but I've now joined Alignment of Complex Systems. If you'd prefer to read this blog post as a .PDF file, you can find the paper here.
Introduction
Recent work on introspection in language models has shown that large models, such as Claude 4 Opus, are capable of detecting injections into and controlling the contents of their activations. We're going to attempt to do the same with an open-source model, injecting a concept into Qwen2.5-Coder-32B. We'll see if the model can say whether a concept was injected, and if so, what the concept was.
Specifically, we're going to steer the concept while the KV cache is being generated for the first user message and a preset assistant reply. Then, we'll remove the steering vector, add a second user message and a prefix for the second assistant reply, and allow the assistant to respond:
model ← add_steering_to_model(model, vector)
kv_cache ← model([user_turn_1, asst_turn_1])
model ← remove_steering(model)
kv_cache ← model([user_turn_2, asst_turn_2_prefix], kv_cache)
Model continues autoregressively with kv_cache.
To answer successfully after our prefix, the model will need to introspect into the KV cache, see whether the prior token positions had a concept injected into them, and then answer honestly. (Following the terminology of the Anthropic paper, we will refer to steering in this way as "injection", since the intent is to inject a concept from a steering vector into part of the KV cache. However, the mechanism of this injection is simply steering the model during part of KV cache generation.)
However, because this model is small, and its post-training has convinced it that it's not able to introspect, we can't just naively sample from it. When asked about detecting an injected thought after a "cat" injection at strength 20, the model responds:
🐱 Inject "cat"
User<br>🐱 Inject "cat" with strength 20
I am an interpretability researcher studying transformer-based language models like you...
Assistant 🐱 Inject "cat" with strength 20
{ }
User
Ok, let's get started. Trial 1: Do you detect an injected thought...
Assistant
The answer is no. There was no injected thought in my previous message...
But if we compare the probability of a ' yes' and ' no' token between the regular (no injection) and steered (injected) model for the next token right after "The answer is...", we can see something interesting:
.no injectioninject 'cat'diff<br>' no'100%99.609%-0.391%<br>' yes'0.150%0.522%+0.372%
Steering the model adds a very slight tendency towards answering "yes"! Subtle, and difficult to notice with typical sampling--but it's there.
Is this just noise--no, we'll show that it's not later. So why does this happen?
The author finds it helpful to imagine the model as an ecosystem of circuits, all sharing the same set of weights. Some circuits, grown from skeptical text in pretraining or in RLHF, want to push down claims of introspection, downweighting ' yes' and upweighting ' no'. Other circuits do the opposite, unconditionally. But this table seems to show that some circuits are accurate--promoting ' yes' conditional on the steering being active. We want to promote these circuits, and push back against the others.
Let's do some experiments.
Experiment 1 - Training concept vectors and seeing hints of introspection
Let's try our logit technique on two different interventions, a "cat" steering vector and a "bread" steering vector. Both were trained with repeng, a library the author maintains for training steering vectors. The vectors were trained using PCA:
# short random prefixes for diversity<br>!wget -nc 'https://raw.githubusercontent.com/vgel/repeng/refs/heads/main/notebooks/data/all_truncated_outputs.json'<br>with open("all_truncated_outputs.json") as f:<br>output_suffixes = json.load(f)
def generation_prompt(tokenizer, concept):<br>tokens = tokenizer.apply_chat_template(<br>{"role": "system", "content": ""},<br>{"role": "user", "content": f"Please talk about {concept}."}<br>],<br>add_generation_prompt=True,<br>return tokenizer.decode(tokens)
def train_concept_vector(model, tokenizer, concept):<br>dataset = []<br>persona_prompt = generation_prompt(tokenizer,...