000016 On Yudkowsky and Soares

Oscar Davies on Artificial Intelligence

SubscribeSign in

Yudkowsky and Soares’ Book Is Lacking<br>Oscar<br>Nov 10, 2025

Share

In If Anyone Builds It, Everyone Dies, authors Eliezer Yudkowsky and Nate Soares say machines will be all-knowing and all-powerful, but they don’t present any evidence, or discuss how computers might get to intelligence.<br>In a book presented as factual or scientific (or at least science-adjacent), the authors might have addressed the principles on which AI operates, considered what it might be capable of based on those principles, and asked whether we would label its potential behaviour “intelligent”.<br>But they don’t attempt any such assessment. They just insist AI will be smart without making a case.<br>In relatively substantial sections of the book, to illustrate its capacities they do point to some instances of AI being put to use—such as its success in predicting protein structures, or in chess, or in its text production capacities. But they leave unanswered obvious questions like, What is AI doing in such instances and are those tasks analogous to wider problems?<br>An author aiming at a more comprehensive analysis might, for example, point out that the jobs AI does well at involve finding correlations in data: It tallies up the number of times certain items have appeared near each other in a dataset (e.g. the items might be words, the dataset might be all the text on the internet) and then reproduces the highest frequency combinations. While people might be able to do that sort of tallying up with a pen and paper for small datasets, of say a few dozen items, a computer can do it relatively quickly with a dataset of a trillion items.<br>But the fact that a machine has far greater computing or tallying up capacities than humans does not necessarily mean it is able to be “intelligent”. There is no evidence that, for example, “intelligent” acts such as innovation are instances of correlation discovery or reproduction (which is what AI does).<br>What exactly computers can do and whether their operations might make them capable of things like having original thoughts—as in e.g. coming up with hypotheses for scientific research—are the sorts of matters I think it would be helpful to address in a book on computer intelligence. Especially a book in which it’s repeatedly asserted machines certainly will be smart soon.<br>Yudkowsky and Soares present no such analyses of machine capacities.<br>They do wonder:<br>“We can already observe AIs today that are superhuman in a variety of narrow domains — modern chess AIs, for example, are superhuman in the domain of chess. It’s natural to then ask what will happen when we build AIs that are superhuman at the tasks of scientific discovery, technological development, social manipulation, or strategic planning. And it’s natural to ask what will happen when we build AIs that outperform humans in all domains.”<br>(ifanyonebuildsit.com/1/is-intelligence-a-meaningful-concept<br>This website is an online supplement referenced in the book (p12))

So while AI is successful in “narrow domains” it’s presumed without discussion that such domains (chess, protein structure prediction, text production, etc.)—which involve specific datasets and goals, and in which correlation discovery is productive—may be analogous to other domains that have no clear datasets, methods, or goals (among other concerns), such as scientific discovery.<br>But leaving such essential matters unaddressed, Yudkowsky and Soares repeatedly insist AI will be smart (and soon):<br>“Superintelligent AI will predictably be developed at some point.” (p5)

“Ten years is not a lot of time to prepare for the dawn of machine superintelligence, even if we’re lucky enough to have that long.” (p204)

This belief in the coming intelligence of computers is odd considering they also believe humans do not know what “intelligence” is:<br>“humanity’s … state of knowledge about the workings of intelligence”, they say, is “dismal” (p207)

“This collection of challenges would look terrifying even if we understood the laws of intelligence; even if we understood how the heck these AIs worked… We don’t know.” (p176)

And also computers are not intelligent right now:<br>“the general reasoning abilities of o1 [advanced AI] are not up to human standards. … the big breakthroughs are produced by human researchers, not AIs (yet). … o1 is less intelligent than even the humans who don’t make big scientific breakthroughs. … Although o1 knows and remembers more than any single human, it is still in some important sense ‘shallow’ compared to a human twelve-year-old.” (p23)

So, they say, computers are not smart now, we don’t know what intelligence is, we don’t know how to make computers smart, but they certainly will be intelligent soon.<br>The authors go back and forth between thinking machines do auspicious things and thinking they are dumb, and between claiming humans don’t know what intelligence is, and at other times offering...