Layer Scope: How I used $20 of compute to make a new way to look at LLMs

Layer Scope: How I used $20 of compute to make a new way to look at LLMs

August 07, 2026

Inspired by Anthropic’s newest paper on LLM interpretability, an excellent blog post series by David Noel Ng, and other research I’m currently working on, I’ve created a cool new way to read the state in the middle of LLM networks! The best part is that it’s a wonderfully simple approach:

Choose a transformer block and token in an LLM you want to inspect.

Run the last 4 layers of the LLM.

With just that I’m able to replicate almost all of Anthropic’s reading examples! I call it the layer scope . If you want to play around with some examples go here.

The rest of the article is here to explain how I arrived at the idea for this probe. I also want to show that AI research is something that’s a lot more approachable than people think. This next part is written more for SWEs who know some things about LLMs and is not intended to be a serious research paper.

A Quick Refresher on high-level LLM architecture

LLMs operate by multiplying your input by an embedding matrix to get a high-dimensional vector representing your token. This vector is known as the hidden state . The hidden state is then fed through a set of transformer blocks, also referred to as layers [1]1. It’s annoying that the word “layer” also overlaps with MLP/feed-forward layers. Whenever I mention layers in this post, just think “transformer block”.. These layers take the hidden state and add some Δh to it. Finally, at the end you multiply by an unembedding matrix to get the final output distribution for the next token.[2]2. Huh, you could write a book on all the stuff I just glossed over… Go check out Welch Labs if you haven’t!

An open challenge in AI/ML is figuring out how to interpret the hidden state as it flows through the model.[3]3. citation needed It turns out that lists of 1000s of numbers make for poor reading material! So let’s go through some of the ways researchers have done this in the past.

Logit Lens

One of the earliest and simplest ways that researchers tried to probe the hidden state between layers was to throw the hidden state directly into the unembedding matrix. This approach became known as the logit lens .

While cheap, simple, and easy to understand, the logit lens falls apart the farther back in the layers you go. Probing the last few layers works alright, but the hidden state seems to be structured fundamentally differently in the earlier layers. This causes the logit lens to start outputting random garbage.

If only there were a way to move from those middle layer states to something that was intelligible to the unembedding matrix… Hey look, a paper by Anthropic about something called the J lens!

What is the J Lens?

The J Lens is short for Jacobian lens. So what’s a Jacobian?

A Jacobian is a matrix of all first-order partial derivatives of a vector-valued function.

In plainer English, it’s a list of derivatives for each value of a vector that you pass into a function. This gives you an idea of how small tweaks to the input vector will change the output of the function.

So what is Anthropic doing here?

(A) Jℓ is computed by backpropagating from the final-layer residual stream to hℓ and averaging the resulting Jacobians over token positions and over a corpus of prompts.

One way to think of this is that we are taking a derivative of the model’s output for the next few tokens with respect to each part of the hidden state vector at a specific layer. This derivative is taken for several of the next tokens and averaged together to get some idea of how things affect future tokens.

The resulting Jacobian is really useful for this specific token in this specific text, but would likely fall apart if we tried to use it elsewhere. If we calculated the Jacobian on a cookie recipe, it might be useful for a cake recipe since they are semantically similar. Try it on a sci-fi story and you might get garbage since the story lives in another part of the hidden state’s vector space not well represented by cooking.

The way the J lens tries to get around this is by calculating the Jacobian for around 1000 different prompts from a large variety of texts (cooking, poetry, math, etc.), and then averaging them together. The hope[4]4. This is a big if. ML research has a long history of things not generalizing, and you could do a research project looking into where this breaks down! is that this results in a more general Jacobian that will tell you the general direction things are going based on the hidden state at a particular layer.

Overall this is expensive! You’re basically doing multiple backprop steps for each of these Jacobians. Specifically that ends up with roughly:

4backprop steps per Jacobian∗1000jacobians∗32layers∗4models2=∼256,000 backprops[5]5. This is divided by two because you only backprop the layers<br>ahead of the layer that you are on. This averages...