No, "AI" will never become conscious — A Lovely Harmless Monster

A Lovely Harmless Monster

No, "AI" will never become conscious

Published on August 4, 2026

In the late 20th century, there was a gorilla named Koko who made waves by displaying an apparent ability to talk to humans in sign language. This would've had shocking implications for life as we know it, but the science determined that what she was doing wasn't actually "using language". Through reinforcement learning, she had come to realize that mimicking the hand signals of her human handlers would result in certain outcomes. She knew that making the hand signal for "food" might convince the humans to give her food, but she didn't have the brain structures that would allow her to understand that the hand sign represented food in an abstract way. She could learn word associations, but she couldn't understand the grammar and syntax required for the communication to actually be a "language". We did not discover Gorilla sapiens.

That's not to say it's impossible; whatever mutation made human brains develop language could happen again, and if it does, we should take it seriously. I don't think this will happen the way it appeared with Koko; we'll probably notice an animal communicating in a slightly more complex way, but it might take eons before the communication becomes something we recognize as language. Or we'll discover a heretofore unknown species of animal that has already undergone this transformation. Extremely unlikely, but it could happen.

So what about the opposite? If a conscious animal can't spontaneously learn to use language, can a language-using being develop consciousness? Some people think it's possible; some claim to believe it's already happened, that currently-existing LLMs are already conscious. This is a possibility I'm more skeptical about.

First of all, I dispute the idea that LLMs are "using language" at all. They're doing something that looks, to a casual observer, a lot like using language; but lest we forget, so did Koko. The way LLMs learned language is basically a high-speed, highly specialized version of the way Koko "learned" a "language". We made a computer program simulate an animal that "wants things", then taught it to mimic human language in a way that "gets it what it wants". If you had a virtual Koko whose brain was the same but worked a trillion times faster, she might also be able to "learn language" to a similar degree.

Ted Chiang wrote the definitive debunking of AI consciousness, so I won't rehash his analysis, go read it. (if the full article won't load, try disabling javascript.) But I think continuing the animal comparison will illuminate why "AI consciousness" is not only impossible, but a concept that only makes sense in a religious framework.

We don't know exactly when or how consciousness first appeared, but we do know why, because the "why" of evolution is always the same: consciousness evolved because it helped an animal survive. Some organism mutated some feature resembling a prototypical brain, with some primordial spark of awareness; this (somehow) helped it survive and reproduce in greater numbers than the purely stimulus-based animals around them. Over millions of years, the organism kept outcompeting the ones in the niche without this mutation, and as they kept reproducing, the mutation kept mutating. Brains got bigger and the spark grew along with them.

This is how evolution works in the natural world: (mutations + suitability)*(time) = [differences]. The most useful differences within a particular niche become the dominant ones.

Evolution is a useful metaphor for machine learning, because some concepts from evolution map neatly onto self-improvement algorithms. However, this metaphor has become load-bearing for the AI industry, whose existence depends on making sure we confuse the map for the territory forever. If self-improvement algorithms are sort of like evolution, and computers can "evolve" a zillion times faster than organic life, then AI will probably "evolve" into SUPER AI within the next 5 years, or if not, then definitely the next 10, trust me bro.

The problem is that a metaphor is only a metaphor. In reality, machine learning is different from evolution in important ways.

Have you ever played with the Genetic Algorithm 2D Car Thingy? If not, check it out, it's a fun little toy. Amorphous masses of circles and polygons appear on the left. The circles rotate clockwise. If a circle touches the ground, it will propel the mass of polygons it's attached to forward, until gravity and the shape of the terrain forces it to stop.

In round 1, most of the masses don't get very far. Maybe one or two will get lucky and be generated with two wheels touching the ground. These will go the farthest, but still quickly fall over because their shape is random, not optimized for traversing the hills and valleys in front of them.

Once all the "cars" stop, the universe resets. A new batch...