The Experience Feedback Loop | peterbloem.nl
The Experience Feedback Loop
I’d like to share something here that has given me a small amount of hope recently. For a long time, the onslaught of AI has made me quite depressed and unmotivated in my work. Apart from all the immediate problems it’s causing, and apart from the unethical and unlikeable practices on which it’s being built, there is the bigger question that hangs like a storm cloud on the horizon. What will we be good for if this continues? What’s the point of doing anything, if AI is improving at this rate? This is not discussed too much, except in fields like mathematics, where the hailstones are already flying. That could be down to the uncertainty, which makes it difficult to say anything meaningful, but I think denial probably plays an equal part. It’s not an easy question to face up to.
This is a long, rambling essay, so I’ll give you the lede up front: hope lies in things that AI cannot do and will never be able to do. It will not do to look to things like novelty, creativity or even intuition. AI can do these things already, and is steadily improving. However, there is one thing that AI still struggles with and, with a bit of luck, will continue to struggle with for a long time. That is experiencing the world as a human being. We are odd little creatures and the way we perceive the world is largely made up of little accidents. You cannot just mimic that by getting ever smarter. And experiencing the world like a human being is required for most activities that we value intrinsically.
Some people like to say that no machine can ever do X or Y, because there is something special or distinct about the human brain. Sometimes this is explicit, but more often, it’s an implicit assumption. When you ask mathematicians how their job will change in the age of AI, they will often tell you that they will be in charge of the general directions—the research taste—while AI will deal with the details. Behind that picture of the future is a hidden assumption that AI will hit a limit: it will be better than us at proving well-stated theorems, but it will never be better than us at coming up with which theorems we should try to prove in the first place. It can execute a plan we give it, but it can never design a plan of its own as well as we can, or set out a fruitful general direction of investigation. In short, AI has fundamental limitations compared to human beings.
I don’t believe that. I have studied AI since 2001 and I have always been a strict materialist. I believe you could simulate the brain in a computer and it would do exactly what we do. The question has only ever been how faithfully you need to simulate the brain. Does it need to be perfect down to the atom, or will a crude approximation do? Given how robust the brain is to damage, my guess has always been “not very faithfully at all,” and recent developments are bearing that out.
Other people will say that modern AI is heading down a cul-de-sac. It will hit a wall, none of it will pay off, it’s all a big parlor trick. This discussion has been going on since deep learning came along over ten years ago, and by and large, AI capabilities have grown exponentially while the critics feverishly move their goalposts. Nothing I have seen over the past 15 years tells me that we are moving in the wrong direction, away from achieving a general, human and super-human level of artificial cognition.
Certainly, modern AI systems like Claude or ChatGPT are missing some basic aspects that we would associate with “full cognition.” Things like persistent memory and real agency, but these limitations are to some extent by design. They’re not limitations that could never be resolved, and what’s more, the parts that we haven’t provided modern AI yet are much simpler to build than the parts that we have provided them. The main reason that we haven’t built a fully intelligent agent yet, I think, is that we don’t really want to. It would be expensive and hard to control, without a huge payoff. But the incentives are constantly shifting towards giving AI more control and more agency.
As for persistent memory, it turns out that some AI will happily make its own, even when we really don’t want it to.
This, then, is my perspective. Intelligence is not specific to the human brain, and in fact we are finding that it is unbelievably, eye-wateringly easy to create in silico. It’s much easier than it was to, say, go to the moon, to split the atom, or to fly across the atlantic ocean. And the intelligence we are creating, rather than staying at our level, is shooting past us left, right and center. We are about to go from the smartest entities in our known existence to “nothing special”. Maybe in a decade, maybe before the end of 2027. Quite apart from the social impact that that’s going to have, I can’t help but think it will have a profound psychological impact as well.
So where’s that good news I was talking...