What policy makers need to know about AI safety and security
What policy makers need to know about AI safety and security
Posted by ekr on 26 Jul 2026
We can all see that AI is rapidly changing the world, and it's<br>bringing with it a whole new set of safety and security issues.<br>Fundamentally these are technical problems, but they raise enormous<br>policy questions about how to manage the risks associated with these<br>new technologies. As with all technology policy questions, making good<br>decisions requires some understanding of the technology, but it's<br>also not practical for policy makers to read and understand<br>what is an enormous and often highly technical literature. Rather,<br>technologists need to find a way to turn that literature into<br>something that is readily digestible while also retaining the<br>key points that you need to understand in order to make good<br>decisions.
Over the past half year or so I've been working on a white paper that<br>tries to do just that, which I'm calling "What policy makers need to<br>know about AI safety and security" where I'm finally prepared to release<br>as a working draft. This isn't a finished<br>artifact (hence "working draft") but has seen some level of external<br>review and so is hopefully good enough that people will find it useful.<br>If you find mistakes, please let me know.
For those of you who don't want to read a 39-page PDF, here is the<br>Executive Summary:
Artificial Intelligence (AI) is rapidly changing the world.<br>AI is an incredibly powerful set of technologies that<br>offers a new and remarkable set of capabilities. Like most such technologies,<br>it also comes with serious new risks. Even under non-adversarial conditions,<br>AI systems can misbehave badly, and attackers can construct input to create<br>specific attacker-controlled results. Moreover, AI is a dual use technology<br>which can be used directly by attackers to generate misleading, undesirable,<br>or dangerous content.
Existing models are already capable enough to be<br>useful to attackers. They can<br>mount sophisticated cyber attacks, create convincing fake content, and<br>make realistic explicit content of children and nonconsensual third parties.
It is not practical to prevent model misuse by sophisticated users.<br>Model guardrails are modestly effective for hosted models, although<br>there is an ongoing arms race between attack and defense. Sophisticated<br>users can use open models, which have unconstrained behavior and are<br>already powerful enough for many applications. In order to manage<br>the risk of misuse of these models it is necessary to focus on the<br>downstream impacts of that misuse.
It is difficult to prevent the distribution of malicious AI-generated content.<br>Once malicious content is generated, it can be disseminated through<br>existing channels. Much such content cannot be reliably detected<br>at all, and even in cases, such as AI-generated child sexual abuse<br>material (CSAM), where some detection is possible,<br>it is still imperfect. It may be possible to mitigate the risk of<br>AI-based disinformation by improving mechanisms for demonstrating<br>the provenance of legitimate content.
AI models exhibit unpredictable behavior. Even in<br>non-adversarial situations, they have the potential to misbehave and<br>cause serious damage. In adversarial situations, the risk is<br>even higher; an attacker who can provide input to the model<br>can deliberately trigger misbehavior. It is not presently<br>known how to eliminate this risk entirely.
Using AI models requires trusting the model provider. Using<br>hosted models requires sending data to the model provider and trusting that<br>it will not misuse it and that it will provide appropriate answers.<br>Local models are somewhat safer, but may still behave<br>maliciously if they were trained by untrusted third parties.<br>Ensuring the existence of both hosted and open weight models developed<br>by trusted providers is critical for reducing the risk of malicious<br>models.
AI models are a powerful dual-use cybersecurity tool.<br>AI models are very effective at finding and exploiting vulnerabilities<br>in software. While there has been some progress in the<br>use of AI to secure software systems, the current situation favors<br>attackers and may continue to do so for some time.
It is not known how to completely secure agentic AI<br>systems. Any time an AI system is exposed to potentially malicious<br>input—which is to say any input from a third party—its<br>behavior becomes untrustworthy. Current best practice is to assume<br>that the model will misbehave and attempt to limit the damage it can<br>do. This applies even in non-adversarial settings because of the<br>unpredictability of current models.
AI technologies are far too powerful to abandon and we should not want<br>to. Moreover, even if advancement in AI models proper were to cease today—which<br>there is no evidence will happen—we are nowhere near finished exploring<br>the new capabilities that current models offer, which means that we should<br>expect to see continued deployment of new AI systems, both by legitimate<br>and...