Responding to AI Distillation Without Panic | Lawfare
The upcoming main navigation can be gotten through utilizing the tab key. Any buttons that open a sub navigation can be triggered by the space or enter key.
Search Lawfare
Search
Advanced Search
Bahrad A. Sokhansanj
Meet The Authors
Subscribe to Lawfare
Chinese large language model (LLM) developers are under scrutiny for reportedly employing large-scale “distillation attacks” on U.S. frontier artificial intelligence (AI) models to improve their own systems. Many U.S. actors have sent signals that they consider distillation a serious threat. For example, in May, Anthropic released a policy paper during President Trump’s trip to China, highlighting distillation attacks as a key challenge in U.S.-China competition. In April, the White House issued an official memorandum about distillation, warning about “deliberate, industrial-scale campaigns” from Chinese entities. Also in April, the House Foreign Affairs Committee universally advanced a bill called the Deterring American AI Model Theft Act to address the issue. And others have circulated additional policy proposals.<br>Discussions of distillation often take for granted that it is a form of theft. But there are key differences between “stealing an AI model” and distillation that policymakers should recognize. To properly address distillation, policy should focus on illegitimate model access—and avoid imposing poorly targeted rules that could harm Americans and distort the open and competitive U.S. AI ecosystem.<br>What Is Distillation?<br>The concept of distillation has evolved since it was introduced as a machine learning technique in which a larger “teacher” model’s outputs are used to train a smaller “student” model. Traditionally, that often meant training the student model on the teacher model’s probability distribution over possible outputs, rather than only on the correct answer. Today, the term is used more broadly. “Distillation” also includes prompting a frontier model to generate outputs, and then using the prompt-output pairs—or, where available, reasoning traces—as training data to refine a model. Frontier models may also be used as judges or verifiers for reinforcement learning. Together, these methods improve weaker models by training on stronger models’ responses to prompts and solutions to complex problems.<br>Distillation is a common practice in contemporary AI development. While on the witness stand at the recent Musk v. Altman trial, Elon Musk acknowledged that xAI had done at least some distillation of OpenAI models and that “generally AI companies distill other AI companies.” As Nathan Lambert, a leading U.S. open-source AI researcher, recently wrote, distillation helps train smaller, often open-source or open-weight models. The White House has recognized this: Office of Science and Technology Policy Director Michael Kratsios pointed out that “AI distillation, when legitimately used to produce” such models, is a “vital part” of creating open models and ensuring a competitive AI ecosystem.<br>But some Chinese AI developers appear to be using distillation well beyond ordinary practice, accessing U.S. frontier models at a massive scale to do so. In February, Anthropic reported that three Chinese AI labs had generated more than 16 million exchanges with Claude through approximately 24,000 fraudulent accounts, in some cases using jailbreak prompts to extract as much information as possible. OpenAI and Google have also reported or detected similar distillation efforts.<br>The unusually aggressive distillation efforts of Chinese labs have been portrayed as an attempt at “model theft” and to “steal” the intellectual property of frontier AI labs. But while calling distillation a form of “stealing” or “theft” may make for effective rhetoric, it isn’t an accurate description of how distillation of a closed AI model really works.<br>Why Isn’t Distillation “Model Theft”?<br>Distillation doesn’t involve breaking into a developer’s internal system to download the model weights or source code. To a distiller, the model is still a black box. In this context, then, “model theft” would mean some kind of black-box extraction—learning enough about a model from the outputs to approximate model behavior such that it effectively steals the developer’s intellectual property (IP). But what IP would that be?<br>To start, copyright can be ruled out. The aspects of a model that could plausibly be protected by copyright, such as software code, can’t be copied by distillation. Nor should copyright be used to create a backdoor property right in model outputs. An AI system cannot be an author, and AI-generated outputs are protected only when sufficient human authorship is...