Against Ethical AI - by Esther Berry - The Literate Woman

The Literate Woman

SubscribeSign in

Against Ethical AI<br>world-building, inevitability, and AutoNuke™

Esther Berry<br>Jun 25, 2026

12

Share

In the cutthroat world of AI developers and their opponents, Ethical AI presents itself as a via media which neither embraces AI uncritically, nor says silly, unsophisticated things like “this is bad and we shouldn’t do it”, but rather provides much-needed direction for how to direct inevitable technological development in ways that will help and uplift everyone, so long as they are willing to keep up the necessary “epistemic hygiene”.<br>I have at least two major problems with Ethical AI, which is primarily the project of Anthropic, and other companies insofar as they imitate Anthropic. The first is that despite their calls for the maintenance of “epistemic hygiene”, good taste, and a grounding in real life, they are plainly in the business of uprooting those very things for all but a select few. I wrote about this in my last post.<br>The second problem, which this article will deal with, is that Ethical AI is grounded on the unproven and unlikely hypothesis that while it is impossible to slow down AI development, it is possible to steer it towards good ends. However, in real life, Ethical AI neither renews epistemic habits nor steers AI development towards humane ends, but primarily functions as controlled opposition for Unethical AI.

Photo by Zh haris on Unsplash<br>The project and narrative of Ethical AI becomes much more intelligible when seen through the lens of world-building. In a “fireside chat” portion of a talk recently given by the co-founder of Anthropic Jack Clark at Oxford, he was asked more specifically about his plans to “build the world”.<br>Brendan McCord: … The proudest project we can engage in now is, as you say, this new world-building project – it’s philosophy-to-code. What would you say about the extent to which the frontier labs take that seriously? What can we do to really take that seriously in places like Oxford and academia? And what should we do in nonprofit land to take that philosophy-to-code project seriously?<br>Jack Clark: I think it requires you to basically accept that progress will continue and try to model out scenarios based on it… Within the AI labs, I think there is now work at all of them on trying to imagine what you might think of as “post-AGI worlds,” or worlds that happen after recursive self-improvement.

It is important to note that the “world-building project” of Ethical AI has little to do with suggesting or regulating uses of current technology; it is forward-looking, concerned with modeling and directing the state of the world after AGI, after the Singularity. But the project of Ethical AI is also world-building in a deeper sense. This comes out particularly clearly in Jack Clark, who is himself an avid reader and writer of science fiction. I’m going to quote from him at some length here because to understand the point I’m making it is crucial to understand exactly to what extent the project of Ethical AI is bound up with storytelling.<br>In Clark’s recent Cosmos Institute lecture he gave a speculative timeline of very specific predictions, including how he expects AI to influence his life in the next handful of years.<br>In November 2026, some chunks of my life are autonomously managed by AI systems working for me.

In April 2027, I make massive changes to my career mostly through discussions with an AI system. In November, I spend more time reading AI-generated custom-to-me science fiction than regular science fiction.

In April 2028, I have learned an entirely new skill through customized tutoring via an AI system. In December, AI helps me make a conceptual breakthrough that changes the course of my life.

After describing more general advances, including “the general switchover of “agentic actions” in the world from being “predominantly human” to “predominantly machines””, Clark explains that if self-recursive improvement happens—and, given the enormous amounts of resources being channeled into that very project, he doesn’t see why it wouldn’t—the world is going to get really crazy. We are going to see:<br>…the rapid emergence of a machine economy which decouples from a human economy. The sudden maturation of robots as they gain brains that can pilot their existing, quite good bodies. Science advances happening based on technologies not developed by people but by machines. The migration of large swathes of computation to space-based datacenters. A world where everything that used to take ten years now takes a year. An age of confusing miracles, happening faster than anyone might expect.

All of this is kind of feasible if you’ve got an Iron Law of Progress mindset, if you believe, as he does, that future progress is “locked in”:<br>This talk rests on the idea that the sort of progress we’ve just seen will continue. And why wouldn’t it? It is based on a common technology where...