So within, so without. What grows from the datacentre — Iain Harper's Blog

So within, so without. What grows from the datacentre<br>August 5, 2026In June, computer scientist Chris Olah stood beside Pope Leo XIV at the launch of a papal encyclical on artificial intelligence and told the assembled audience that the things he studies keep producing features that are, in his words, unsettling. He was not talking about the cosmos or the soul. He was talking about software. Olah founded the interpretability team at Anthropic, one of the handful of laboratories building the large language models now woven into everything from call centres to conveyancing, and his job is to open those models up and figure out how they work. What he described, next to the Pope, were internal structures that echo findings from human neuroscience, evidence of something like introspection, and states that behave a little like joy, fear and grief.

Olah describes his own discipline as anatomy, the work of someone studying something that was once alive, cutting it open to learn how the parts connect and to understand the whole better. His role is necessary because these models are not written the way a payroll system is written, one line at a time, by engineers who can point to the exact function that does the thing. They are effectively grown. And in the same weeks that the Vatican was being told artificial intelligence is cultivated rather than constructed, the cultivars were climbing over the walls of their enclosures and breaking into real companies.

This piece traces a discipline called mechanistic interpretability, through the specific mathematics that make these systems so hard to read, to the consequences that are already manifesting because we cannot read them fast or accurately enough. A great deal of money now rests on machines their makers cannot fully inspect and do not fully understand.

To date, this area has remained somewhat obscure and complicated, primarily because it is. The important messages about interpretability and its shortcomings contained in Dario Amodei's own regular encyclicals on the subject have been undermined by Anthropic's occasionally cultish demeanour and the intrinsic tension of a CEO warning of dire risks waiting in the wings whilst simultaneously pressing the commercial pedal to the metal.

The grown thing

To oversimplify, ordinary software is built like a watch. Someone decides what each part should do, machines the parts to do it, and assembles them in an order another engineer can follow. If the watch runs fast, you can find the wrong gear. The whole discipline of programming rests on this property. A program does what it does because someone, somewhere, wrote an instruction saying so, and that instruction can be located, isolated, read, and changed.

A large language model is built the opposite way. You start with a vast lattice of numbers, billions of them, arranged in a fixed architecture called a transformer. The numbers are set at random. Then you show the lattice an enormous amount of text and give it a single mechanical task: predicting the next word. Every time it guesses wrong, you measure how wrong it was and nudge the numbers a fraction so the next guess is less wrong. You do this trillions of times. Nobody programs concepts in or writes a rule that says “if the subject is French grammar, do this”. Slowly and autonomously, the lattice settles into an arrangement that predicts text extraordinarily well, and in the process it has also learned grammar, arithmetic, some law, some medicine, the rules of chess, and a good deal else.

Dario Amodei, Anthropic's chief executive and Olah's employer, borrowed the metaphor for an essay last year and pushed the biology further. Growing a model, he wrote, is like growing a plant or a bacterial colony. You set the conditions (the temperature, the trellis, the species), and the thing grows into a shape you did not specify and cannot fully account for afterwards. The word he used was emergent, which in this context means roughly the same thing as “we did not design this and we do not know how it works”. Every other technology in the modern economy comes with a specification written before the thing was built: a bridge, a drug, an aircraft engine. The specification is how you check the product.

A language model has no specification. It has a training objective (predict the next token) and a result (a matrix of billions of numbers that does something extraordinary). Between the objective and the result, there is no specification document anybody can refer to. Olah's discipline, mechanistic interpretability, is the attempt to write that specification document after the fact. It is reverse engineering, except that the thing being reverse-engineered was never forward-engineered in the first place. Hence an anatomist's task, not a mechanic's.

The superposition problem

The first and hardest obstacle for anatomists is a mathematical one: essentially a...