J-Space Oddity: Do VLMs Dream of Text Tokens? | Blog | Yogesh Kumar -->

🌙 ☀️<br>Home • Blog

Contents<br>&times;

Do vision-language models think in words?

Anthropic’s new paper is a fascinating read on interpretability, high-dimensional geometry, and the art of carefully side-stepping the “is AI conscious?” dialogue. Their experiments, however, stayed text-only, apart from a single prompt made of ASCII art of a face. I wanted to understand if their method also generalizes to VLMs. And if yes, would it work with a Jacobian fitted on text-only prompts?

Nothing forbids this really, a VLM like Qwen2.5-VL simply chops an image into patches and buses them through the same residual stream as words. Image tokens just arrive with geometry attached: every patch gets a 2D coordinate, and attention lets it soak up context from its spatial neighbors.

The J-lens

The probing gadget is called Jacobian lens (J-lens), which takes the residual state hℓh_\ellhℓ​ at some middle layer ℓ\ellℓ and asks the question, “if I nudged it a little, how would the final layer’s state hLh_LhL​ move?” That sensitivity is indeed a Jacobian:

Jℓ=∂hL∂hℓ.J_\ell = \frac{\partial h_L}{\partial h_\ell}.Jℓ​=∂hℓ​∂hL​​.<br>Pairing this with the unembedding matrix gives us the “readout” (WUJℓ hℓW_U J_\ell\, h_\ellWU​Jℓ​hℓ​), a score for every word in the vocabulary.

Since we are working with activations, JℓJ_\ellJℓ​ would be a different matrix for every input. The paper integrates out this dependence by averaging over a dataset of ordinary prompts and crediting each state to every future position it influences. Writing hℓ(p)h_\ell(p)hℓ​(p) for the residual-stream state at layer ℓ\ellℓ and position ppp, the lens is

Jˉℓ  =  1N∑i=1N⏟prompts    1∣P∣∑p∈P⏟positions    ∑p′≥p1∣P∣∑p⏟future    ∂hL(p′)∂hℓ(p)\bar{J}_\ell \;=\; \underbrace{\frac{1}{N}\sum_{i=1}^{N}}_{\text{prompts}}\;\; \underbrace{\frac{1}{|\mathcal{P}|}\sum_{p \in \mathcal{P}}}_{\text{positions}}\;\; \underbrace{\sum_{p' \geq p}\vphantom{\frac{1}{|\mathcal{P}|}\sum_{p}}}_{\text{future}}\;\; \frac{\partial h_L(p')}{\partial h_\ell(p)}Jˉℓ​=promptsN1​i=1∑N​​​positions∣P∣1​p∈P∑​​​futurep′≥p∑​∣P∣1​p∑​​​∂hℓ​(p)∂hL​(p′)​<br>Read it right to left : a sensitivity matrix, credited to every future position, averaged over positions and prompts.

At the end of all this, we are left with one frozen linear map per layer. If I point this frozen lens at every layer and every position, each state reads out as a small handful of words. That whole collection of unspoken words is what the paper calls the J-space. Anthropic frames it as the model’s “workspace”, sorta like an inner monologue.

Setup

Once again I dust off my workhorse RTX 4090 and run Qwen2.5-VL-7B-Instruct (28 decoder layers, hidden width d=3584d = 3584d=3584) in bf16. I fit the J-lens on a pretraining-like dataset, the same way Anthropic’s paper does, just at a smaller scale. I sample 100 text-only prompts from wikitext and compute the Jacobian matrix (Jˉℓ\bar{J}_\ellJˉℓ​) for each layer ℓ\ellℓ. Everything collapses into one 3584×35843584 \times 35843584×3584 matrix per layer: one lens for each of the 27 layers that feed a later one, since the final layer is the readout itself. On the 4090 that’s about 51 seconds per prompt with the backward passes batched eight rows at a time, roughly 85 minutes end to end, and about 21 GB VRAM. What comes out is a frozen stack of 27 matrices that has only ever read Wikipedia.

Aside: how converged is a 100-prompt lens?Anthropic’s paper fits the lens on pretraining-like dataset of 1000 prompts. Since I am GPU-poor, I had to scale it down to 100 prompts.<br>How converged is a 100-prompt lens? The lens is a Monte Carlo average: each prompt contributes one Jacobian per layer, and we keep the mean (measured here on a random 128-row subsample per layer). Left: distance from the running mean over prompts 1–50 to an independent reference, the mean of the held-out prompts 51–100. If prompts were iid draws with per-prompt scatter σ\sigmaσ, this distance would follow σ1/n+1/50\sigma\sqrt{1/n + 1/50}σ1/n+1/50​, a 1/n1/\sqrt{n}1/n​ decay that bottoms out at the reference’s own noise; the dashed envelope spans the least (layer 0) to most (layer 26) stable layer. Right: the same σ\sigmaσ cashed out as error bars on the fit itself: predicted error σ/n\sigma/\sqrt{n}σ/n​ at n=100n{=}100n=100 and n=1000n{=}1000n=1000, against the measured error (distance between two independent 50-prompt lenses, halved to convert to the n=100n{=}100n=100 scale). Measured runs ~1.5× above predicted (per-prompt Jacobians are heavy-tailed) and error falls steeply with depth: the layers the narration maps actually read (21–26, shaded) are estimated to within 2–9%, and a 10× bigger overnight fit would only improve that by a further factor of ~3.

Experiments

With the lens polished and ready, probing takes a single forward pass. We hand the model one sequence: a question like “Describe this image.”, followed by the image, which the vision tower has already...