Superintelligence is a dragon

Skip to content

Sign in<br>Subscribe

This is a column about AI. My fiancé works at Anthropic. See my full ethics disclosure here.<br>On Sunday night, HBO aired the third-season finale of House of the Dragon, the stuffy and unrelentingly bleak prequel to Game of Thrones. The show takes place in a world where one great house holds a near-monopoly on a superweapon — dragons — and thus rules the world. House of the Dragon begins when the ruling dynasty splits and both sides retain access to their dragons, upending the balance of power in Westeros for the first time in generations. In this world, when only one global power had access to a superweapon, the result was a brutally enforced peace. And when two global powers gain access to the superweapon, the result is a ruinous civil war.<br>In Westeros, no one is confused about what a dragon is; in all likelihood one has been responsible for the death of someone they know. But here in our world, where the technology industry is investing trillions of dollars into building and serving ever-more powerful artificial intelligence models, we struggle to imagine the end state. This is not entirely our fault: while we are in the midst of a rapid improvement in model capabilities, day-to-day life feels more or less the same. The CEOs I speak with on the subject mostly cannot even imagine that a true superintelligence will result in significant job loss, much less greater harm. We fixate on the idea that AI will cure all disease, and mostly ignore headlines about scientists using AI to engineer new viruses. (Fortunately, these ones pose no threat to humans.)<br>For all the travails of the Westerosi, at least there, a dragon always looks like a dragon. In the real world we are not so lucky.<br>I thought about all this Monday morning while reading “The Future is for Everyone,” Mark Zuckerberg’s facile, self-serving manifesto about “the path to a positive AI future.” Over 6,500 words, Zuckerberg attempts to rally people around the flag of artificial intelligence, based on its power to give people new creative, business, and personal assistance, and aid with medical and scientific research.<br>“Invention, not automation, will be the greatest contribution of superintelligence,” Zuckerberg writes. “Early AI could answer questions and do routine work. Soon it will increasingly help discover new knowledge — ranging from discovering new drugs to cure a family member's disease to finding new ways to improve your business.”<br>For businesses, invention and automation go hand in hand, and it remains uncertain what net effect this will have on jobs. On those other points, though, Zuckerberg is correct. AI will give people incredible new creative tools, and help them organize their lives, and drive academic disciplines forward. (Today Anthropic published a blog post claiming that an unreleased model made substantial progress on a problem related to the 167-year-old Riemann hypothesis after an employee with no formal mathematical training asked it to and told it to believe in itself.) To the extent I feel optimism about AI, it is because of these things.<br>Zuckerberg is a relative latecomer to AI boosterism. OpenAI CEO Sam Altman wrote about AI’s potential for positive social change in a 2021 essay called “Moore’s Law for Everything”; he followed it up in 2024 with “The Intelligence Age.” Weeks later, Anthropic CEO Dario Amodei published “Machines of Loving Grace,” his own account of how AI could change the world for the better.<br>Altman and Amodei’s essays were notable because both men had spent years warning about AI’s potential to be destructive. In 2015 Altman called superintelligence “probably the greatest threat to the continued existence of humanity.” And in January of this year, after sounding similar warnings for years, Amodei published “The Adolescence of Technology,” laying out in detail various nightmare scenarios in which we either lose control of AI or power over it is seized by an autocrat.<br>It is against this grim backdrop that Zuckerberg seeks to define himself and Meta. “It is surprising that the discourse from many developing AI is so filled with doom,” he writes. “I do not understand why anyone who believes that AI will eliminate most jobs and much of humanity's relevance would rush to build that future.”<br>What follows is a series of policy proposals that map neatly to Meta’s commercial interests. He writes that we should accelerate the process for building data centers, which may help the company build its nascent neocloud business. We should maintain export controls on advanced chips, which advantage Meta’s open-weights models over its Chinese rivals. We should reduce “training data restrictions,” which will help Meta fight ongoing lawsuits from the creatives whose works were used in creating its models. And we should create legal protections for distillation, which lets Meta draft off the innovations of the frontier labs by training its models on their...