Contra Chiang on machine consciousness - by Henry Shevlin
SubscribeSign in
Contra Chiang on machine consciousness<br>If you think you understand AI consciousness, then you don't understand AI consciousness
Henry Shevlin<br>Jun 17, 2026
40
12
Share
See also Bentham’s Bulldog, “Ted Chiang Is Wrong About AI Consciousness”, and Rob Wiblin, “Ted Chiang is Wildly Overconfident About AI Consciousness.”
Ernst Haeckel, "Hexacoralla," Kunstformen der Natur (1904)
Ted Chiang is one of my favourite authors. I was a philosophy PhD student when I first came across his novella The Lifecycle of Software Objects back in 2011, and it completely blew me away in its nuanced treatment of the identity, moral status, and autonomy of advanced (and possibly sentient) machines.<br>By contrast, I’ve found his non-fiction writing on contemporary AI systems quite frustrating. Shortly after the release of ChatGPT in early 2023, for example, in a comment piece in The New Yorker he characterised the model as a “blurry JPEG of the web,” and suggested that LLMs could at best provide paraphrases of information that was already out there, with limited utility for true creative work. While the piece correctly anticipated some things, including the torrent of slop that LLMs have enabled, other predictions haven’t held up.<br>Most notably, Chiang predicted that labs would carefully exclude LLM-generated text when training future models, on the grounds that training on model outputs would be like photocopying photocopies. As it happened, within a couple of years LLM-generated outputs were being routinely used for both pre- and post-training of frontier models, with painstaking benchmarking showing that, used appropriately, they can improve performance.<br>More broadly, the blurry JPEG analogy was an unhelpful one, and anyone who anchored on it too strongly would have been poorly placed to anticipate what happened next, from reasoning models and IMO gold medals to novel mathematical proofs. A JPEG is a static and compressed artefact; LLMs, by contrast, are deliberative and generative systems capable of delivering novel insights and analysis and producing the very data that improves their successors.<br>Chiang’s AI skepticism has recently spilled over into the domain of consciousness, very much my home turf. In a new signed essay in The Atlantic, he confidently declares that contemporary AI systems are not conscious, and insists that suggesting otherwise is an error of “titanic magnitude.” These systems are just “cleverly disguised examples of sentence continuation,” and that to even be open to the possibility of consciousness in LLMs is “the same as being open to the possibility that Microsoft Word is conscious.”<br>These are extremely strong claims to make about a phenomenon as confusing, complex, and contested as machine consciousness. While I think it’s unlikely that current models are conscious (at least in any form meaningfully similar to humans or animals), I certainly don’t think it’s a basic error or confusion to suggest otherwise, and I’m in good company among cognitive scientists and philosophers of mind.<br>So does Chiang bring the receipts to back up his bold claims? Unfortunately, I think the answer is a pretty clear “no”, and he instead serves us up a buffet of rather stale and in some cases outright confused arguments.<br>Chiang’s essay doesn’t follow a traditional philosophical argument, but I’ve grouped his claims about machine consciousness into four main clusters, none of which come close to supporting his bold headline, as we’ll see. I have a bit more sympathy for some of the ethical and political musings he closes with, and I’ll return to these at the end of this piece, but even there I think he gets way out over his skis and makes some overly strong (not to mention uncharitable) assertions.<br>(1) Most Things Are Lots Of Things
At the core of Chiang’s critique is the idea that we fundamentally misunderstand what LLMs are: he sees users as tricked by their linguistic capacities into thinking that more is going on than meets the eye. Rather than being minds (or even proto-minds), “LLM conversations are cleverly disguised examples of sentence continuation”, a “predictive text game” from a machine that “generates only one word at a time.”<br>I should note that in one respect I agree with Chiang: there are real dangers of anthropomorphising LLMs and treating them as more human than they really are. Contemporary AI systems are anthropomimetic, not truly humanlike, and differ dramatically from us in mechanism, architecture, and embodiment.<br>But this isn’t a royal road to dismissal of AI consciousness. Almost anything can be described at multiple levels. We might variously describe a human baby as a mass of fermions and bosons, a collection of proteins, a conglomeration of cells, a biological organism, the child of Mr and Mrs Jones, or a young Englishman. None of these descriptions are in competition with each other, but operate at different levels of...