DSHR's Blog: Distilling The Moat

Tuesday, July 21, 2026

Distilling The Moat

Whisky Still<br>The original function of a Web server was to respond to queries by revealing the appropriate part of their internal data. This necessarily meant that repeated queries, for example from a search engine's or an internet archive's web crawler, could extract the server's entire internal data. Since the extracted data had been published on the Web, it was not trade secret. It was protected by the publisher's copyright. This has led to many lawsuits, for example against the Internet Archive, Google and others. It is the reason search engines only display "snippets" of the content they collect.

AI companies' intellectual property is their models. They spend vast sums funding the technical and human resources to "train" these models, the racks of GPUs in the data centers, and the hordes of workers labeling images, and having "genuine human conversations" with the nascent model. These expenditures are thought to create a "moat" around the value thus generated, because it would be equally expensive for a competitor to create an equivalent model. It is this moat that supports their extraordinary valuations, despite their lack of earnings.

Below the fold I explain why their moat is very shallow.

Like many Web services, the function of an LLM is to respond to queries by revealing the appropriate part of their internal data, i.e. part of their model. Thus, repeated queries could in principle replicate the model. These models are not published, not protected by copyright law[1], and cannot be patented. Their only protection is as trade secrets. Someone using repeated queries would probably be violating the system's terms of service, but this isn't a strong legal protection. The AI companies are not in a strong position to argue that "crawling" their Web servers is illegal because that is how they created their models in the first place, which they argue is fair use.

It turns out that replicating other models by repeated queries is a standard technique in the industry, called "distillation". For example, Tim Fernholz reportd that Elon Musk testifies that xAI trained Grok on OpenAI models:

On the stand in a California federal court on Thursday, Elon Musk was asked if xAI has used distillation techniques on OpenAI models to train Grok, and he asserted it was a general practice among AI companies. Asked if that meant "yes," he said, "Partly."

And Rebecca Bellan reported that Anthropic accuses Chinese AI labs of mining Claude as US debates AI chip exports:

Anthropic is accusing three Chinese AI companies of setting up more than 24,000 fake accounts with its Claude AI model to improve their own models.

The labs — DeepSeek, Moonshot AI, and MiniMax — allegedly generated more than 16 million exchanges with Claude through those accounts using a technique called "distillation." Anthropic said the labs "targeted Claude’s most differentiated capabilities: agentic reasoning, tool use, and coding."

Distillation works really well because the victim is massively subsidizing the use of its service. If use of the service was extremely profitable, distillation would be unaffordable.

In practice, distillation only replicates a part of the victim model. Companies use this, especially on open-weights models, to produce small, specialized models such as those described by David Berreby in Small AI Models Gain Traction Around the World. But the Chinese distillers accused by Anthropic aren't starting from scratch, they already have a model. All they are trying to do is to replicate some capabilities that their model lacks. So they don't need to extract the whole model, just the relevant parts.

One of the features of the modern Internet that I've been writing about for more than a decade is the security disaster that is the The Internet of Things. Because there are hundreds of millions of "smart" devices exposed to the internet, almost all with known, unpatched vulnerabilities, it is trivial to construct vast botnets to attack Web sites from innocent, unattributable IP addresses. Ian Kelling of the Free Software Foundation reports from the front lines in Our small team vs millions of bots:

To begin with, GNU Savannah, the FSF's collaborative software development system, was hit by a massive botnet controlling about five million IPs starting in January. As of this writing, the attack is still ongoing, but the botnet's current iteration is mitigated. The goal is likely to build an LLM training dataset. We do not know who or what is behind this.

This is an example of an AI company using a botnet to collect training data. Suppose a company were to use this 5M IP address botnet for distillation. MiniMax''s "over 13 million exchanges" would be a little under 3 exchanges per IP address.

To maintain their moat, the AI platforms have to do one of two things:

Implement anti-distillation defenses.

Raise prices enough to make distillation attacks...