AI will make biological extinction risks worse before it makes them better

AI will make biological extinction risks worse before it makes them better | Philosophical Multicore

AI will make biological extinction risks worse before it makes them better

An argument goes: If we don’t build aligned artificial superintelligence, we risk driving ourselves extinct for some other reason. We should rush to build ASI quickly, in spite of the risks—the longer we wait, the more vulnerable we are to extinction from a different cause.

Other than ASI, the biggest extinction risk is synthetic biology. Some lab could (accidentally or on purpose) develop a highly transmissible, 100% fatal super-plague that wipes out humanity.

An aligned ASI could stop that from happening by shutting down dangerous biological research, or by developing advanced countermeasures that stop the spread of deadly infections. So the argument goes: We need to build ASI to save us from non-AI extinction risks.

However, that argument doesn’t work. In the near term, AI will make biological risks worse, not better. AI will accelerate scientific research, which will bring us closer to the level of knowledge necessary to build extinction-level pathogens. And in the long term, the way ASI eliminates biological x-risk is by taking control of the world.

Contents

Contents

In the near term, AI makes biorisk worse

AI can’t control scientific progress unless it controls everything

Low biorisk trades off against high AI takeover risk

Accelerating AI development is not a good way to reduce biorisk

This is yet another illustration of the fact that we don’t know what “aligned AI” means

Notes

In the near term, AI makes biorisk worse

Some people imagine that AI models would accelerate defensive research while refusing to assist with developing bioweapons. This plan has two minor issues and one fatal one.

The first minor issue: Current AI model refusals are not robust, and there are workarounds to get information out of them for people who want to. It’s very hard for AI developers to patch all holes, but the jailbreakers only need to find one.

The second minor issue: Even if the leading AI developer makes their model safe and un-jailbreakable, at least one of their competitors will probably fail at that task.

The fatal issue: It’s not just about what AI assistants can do for humans. It’s that AI accelerates the rate of scientific progress. As state of knowledge improves for humanity in general, it becomes possible for humanity to develop existentially risky pathogens, even if AI does not assist directly. It seems impossible to advance biological science while surgically preserving ignorance on just those bits of knowledge that are required to engineer pathogens.

AI might refuse to participate in gain-of-function research, and that would be better than not refusing. But suppose I’m an evil scientist and I want to develop a 100% lethal airborne pathogen. Here in the year 2026, I can’t do it. Even if I’m on the cutting edge of medicine and biology, I still won’t be able to create the “extinction pathogen”, because that would require a level of scientific understanding that humanity simply hasn’t achieved. If AI advances science in general, it will push me closer to my evil goal of killing everyone with bioweapons.

There is the question of “offense-defense balance”: is it easier to develop deadly pathogens, or easier to protect people against pathogens? That question matters in many contexts, but it’s not relevant here. At our current level of scientific understanding, we have ~zero ability to develop extinction-level bioweapons. If our understanding becomes sufficiently advanced, then that ability will move from zero to nonzero, regardless of the offense-defense balance.

Leaving AI out of the picture, humanity will probably have the knowledge necessary to make extinction-level pathogens within the next hundred years. If AI causes a hundred years of progress in the next decade, then the evil scientist will be able to engineer their extinction pathogen by 2036, thanks to AI—even if the AI itself doesn’t directly participate in the creation of the pathogen.

By 2036, assuming AI hasn’t killed us yet, biorisk will be higher than in the alternative 2036 where AI capabilities stopped improving. Would 2036-biorisk-with-AI be higher than 2126-biorisk-without-AI? Maybe not—maybe AI scientists would be safer than human scientists per unit of research effort. But at minimum, AI-accelerated science is more dangerous per unit of time. AI acceleration means the high-risk period starts sooner, and it means we have less time. Less time to identify risks, less time for policy-makers to respond, less time to consider what direction we should go in. Speedrunning through a century of progress in a decade makes it much harder to manage the risks as they come.

AI can’t control scientific progress unless it controls everything

The only way to accelerate scientific...