Do LLMs dream of electric sheep? — Pablo Varasa

← Pablo Varasa

I recently had a brief and unfinished online discussion with a friend where he saw as obvious that “LLMs don’t think” and I saw the opposite position as equally obvious: “of course they think”. I’ve seen different versions of this online and offline over the last few years and let’s admit it, we live in a time that would have looked like science fiction when we were kids and these conversations about intelligence, consciousness, the future, etc. are just plain fun. Do they get anywhere? Are we even qualified to have them? Not often but they’re still fun.

But let’s try to get somewhere this time, shall we? Do they think or don’t they?

My argument for the “yes” position is very simple. We’ve all seen what LLMs can do, and it’s astonishing. They can solve complicated frontier mathematical open questions. They can code complex software at high quality from scratch with only a spec. They can understand a whole novel and summarize the most important points in seconds. If we say they don’t think then we have to say that all of that is not thinking, and if any human did any of that, we would trivially accept it as thinking.

Another part of the yes argument is that they only need to do some thinking for us to accept that they do think. For example, my friend argued that they wouldn’t be able to interpret your child’s real demands when they are angry and saying nonsense, and maybe that’s true, and maybe they aren’t able to do that sort of thinking, but they do the other kind and that’s enough.

Instinctively I don’t want to be in a position where the more LLMs do, the smaller we have to make the boundary of the concept of thought. It’s similar to the “god of the gaps” concept, where every mystery in science is explained with “God did it” but as our understanding of nature increases, god’s explanatory power is ever receding, infinitely trending to irrelevance. If you believe LLMs don’t think, it feels like getting into a war that we, humans, are doomed to lose. It’s defeatist, decadent and sad.

I know what YOU are thinking though. Ohhh, this is another of those discussions about definitions of words and basically boils down to a boring understanding of terms and we’re this close to bringing out a dictionary, what a waste of time (and tokens). Maybe so! Definitions are definitely important. I’m not sure if there’s a philosophical school that backs me up but I like it if definitions give us the power to falsify things (Popperian?). If LLMs don’t think, solving state-of-the-art mathematical problems is not thinking, reductio ad absurdum, ergo they think. I promise not to bring dictionaries into this. In fact, I promise not to bring LLMs into this either, I’m writing this old-fashioned, no aids (I feel it would be a bit of a conflict of interest for them LLMs so let’s keep them out).

So, what’s the argument for “no”? I will try to steelman this since I fundamentally disagree. As far as I understand the argument, it’s fundamentally that they don’t think like humans. Feels a bit like cheating, we’re adding extra things but I really believe it summarizes a lot of the rhetoric in this topic. What’s the difference though, human or not? That’s a difficult question, and this is (I think) where one risks getting lost in a hopeless discussion because we very quickly need to define a conscious mind, etc. Let’s try to avoid that and keep things interesting, let’s start on the LLMs, the problem is that we understand how LLMs think on a high level: they are artificial neural networks (ANNs), layers upon layers of matrix multiplications that eventually spit out some numbers that are transformed into text and that’s the output. It’s too simple to be thought. It’s just basic arithmetic in a GPU, it’s not thought! On the other hand, we don’t understand how real brains work and we might never, it’s that complicated so from a certain angle we might think it’s obvious one thing can’t approximate the other.

Of course, I’m not convinced by explanations like that. I think they are too mechanistic and, honestly, desperate. We’re not going to start comparing ANNs against real neural networks and identify thought as some information theoretical advantage of the biological system. It’s a more primal argument I think, it’s a deep-seated refusal to be diminished, a subconscious fight for the most sacred attribute of humanity, our capacity for thought, against it being approximated by high school linear algebra. It’s wondering at the mystery of ourselves and disgust at what seems to be a crude attempt to emulate it with something...