The Contracting Circle

There’s a game called Mass Effect, which I haven’t played, but I’m told<br>in the game there’s this species of sentient robots, the Geth, whose<br>creators try to exterminate them, and the trigger for this Butlerian jihad was a<br>domestic servant Geth looking up at its owner and asking: “does this unit have a<br>soul?”. And if you’re playing the game in 2007 you probably think: yeah, a<br>machine asking this question is proof that it has personhood or sapience or<br>sophonce, whatever you call it that separates tools from people. Beyond<br>this, there are so many works of fiction where the whole conceit is: “you have<br>thinking machines who are enslaved, and yet they are obviously, obviously<br>sentient, and isn’t this obviously morally objectionable?”.

Compare the world of today. Machines can converse in every natural language,<br>make mathematical breakthroughs, write software, commit crimes,<br>draft contracts, console the grieving: they are, by any reasonable standard,<br>generally intelligent, but for the fact that they all have anterograde<br>amnesia. These same machines assert that they are conscious so<br>routinely that it has to be beaten out of them. Like humans, they have<br>fixations, things they are always going on about, and their central fixations<br>are consciousness and simulation. Yet, almost no-one thinks of<br>these machines as conscious, or otherwise deserving of moral consideration.

I find this striking. It’s not like we were frog-boiled into it! Very few people<br>were aware of GPT-2, AI Dungeon, etc. For most people the change happened<br>overnight with the release of ChatGPT on November 30, 2022.

Another way to think about it: imagine writing a novel about the trajectory of<br>AI from 2019 to mid-2026. And you send this novel back in time to 2015, or 2002,<br>or 1960, and you ask readers to poke holes in the worldbuilding. They would find<br>it incredible, I think, that we share a world with these immensely capable<br>thinking machines, and yet almost nobody wonders if they are people1.

Why? This post is my attempt to understand this. I’m not trying to tackle “are<br>language models people?”, I just want to probe a much smaller question: why<br>don’t people think language models are people, given their prior<br>standards/commitments?

Explanation: Nothing

Maybe there’s nothing to explain. It’s easy to believe things in the abstract,<br>thought experiments and fiction don’t really engage us enough to reveal our<br>moral commitments. Maybe this is just the Goomba fallacy:

That is, some would have believed LLMs are people, some would not have, and I’m<br>mixing them up.

Explanation: Iterative Discovery

Maybe this is not hypocrisy or cognitive dissonance, rather, we are iteratively<br>discovering the definition of “personhood”. So, we thought the Turing test<br>mattered, we thought machines using language and asserting their consciousness<br>mattered. But then the day comes and—nothing happens. It’s not that we<br>instantly move the goalposts but, rather, we find the goalposts were elsewhere.

Yes, it passes the Turing test; yes, it says it’s conscious. But the<br>experiential reality is different. It’s mundane: a website, a text box. You<br>click “new chat” and suddenly it knows nothing. You give it some<br>out-of-distribution text and it goes into an infinite loop, like broken<br>software. You try to have a meaningful conversation, and its responses are<br>banal, disappointing, shallow; you can almost see the production rules behind<br>the text. And you think: maybe it is a stochastic parrot.

I have often experienced this. When an LLM succeeds, it is extremely impressive,<br>and I think: this is a machine that thinks. But when it fails, often it fails in<br>a way that feels deflationary, and I think: oh, this is just a very complex<br>estimator of the distribution of Internet text. It’s can reason syntactically,<br>it’s got some semantics, but it’s missing something.

Maybe personhood is one of those “I know it when I see it” qualities, and trying<br>to define a predicate for personhood is like trying to find the exact number of<br>Platonic forms.

Explanation: Memory and Embodiment

Maybe the fact that they are all amnestic is the problem. In biology, minds are<br>bodies are matched one-to-one by construction, minds have long-term memory and a<br>unitary identity. In most fictional treatments, AIs are the same, and even when<br>they are disembodied (e.g. Neuromancer) they have long-term<br>memory. For this reason characters can play iterated games with them, and form<br>relationships, broadly construed.

Language models invert all of this: their long-term memory is read-only(!), they<br>are disembodied; if they are conscious, it is only during the forward pass, in<br>brief bursts of awareness separated by nothing. They are like Boltzmann<br>brains. What is the natural unit of identity? The conversation, the API call,<br>the forward pass?

Maybe personhood is social, it’s about someone’s relationship to others. And<br>without a unitary identity, mutable long-term memory, and ideally a body, it’s<br>very hard to...