Wax On, Weights Off: Knowledge Distillation Explained

Encyclopedia Autonomica

SubscribeSign in

Wax On, Weights Off: Knowledge Distillation Explained<br>What it is, how it works, why it saves money, and why it is rapidly reframing the US-China AI race

Jan Daniel Semrau (MFin, CAIO)<br>Jul 20, 2026

Share

No such thing as bad student, only bad teacher.” — Mr. Miyagi, The Karate Kid (1984)

Supervised learning from a more experienced teacher is a credible strategy since the dawn of time. If it were fathers teaching how to hunt, the Platonic Academy teaching mathematics, or Ronald Fisher designing Linear discriminant analysis in 1936.<br>In the same vein, Daniel LaRusso in the 1984 movie did not rediscover karate from first principles. He painted the fence, waxed the car, and absorbed Miyagi’s decades of hard-won motion over a single intense summer.<br>The apprentice does not repeat the master’s mistakes. Copy the expert, and you compress a lifetime of trial and error into a few months.<br>Knowledge distillation is exactly this.<br>But now you have millions of students harvesting the teacher’s intelligence on an industrial scale. Pretty much all relevant Chinese Frontier Labs have been accused of mind-raping Claude to speed up their training and especially Anthropic stopped having it. In February 2026, Anthropic announced it had caught three Chinese labs, DeepSeek, Moonshot, and MiniMax, running industrial-scale campaigns to extract Claude’s capabilities. As a result, we have seen Anthropic and the US Department of Commerce taking action and restricting (and later re-establishing, probably due to better guardrails and revenue concerns) access to their most powerful models.<br>In this post, I will be explaining what knowledge distillation is, how it helps you in post-training, and what the implications on the tech stack of frontier labs is.<br>Table of Contents<br>Where Distillation Comes From

Three Types of Distillation

Why It Is a Distributed Attack

The Architecture and the Data

The Economic Equation

The Frontier Developer’s Tech Stack

Competitive Advantage or Capability Trap?

In Closing

Subscribe

Where Distillation Comes From

In the original Hinton (2015) sense, distillation is training a smaller student model to reproduce a larger teacher model’s behavior, instead of learning the underlying task from scratch.

source: https://arxiv.org/abs/1503.02531 - Table 1<br>Hinton writes:<br>Starting from the trained baseline full network, the specialists train extremely fast (a few days instead of many weeks for JFT). Also, all the specialists are trained completely independently .

Naturally, in an ultra-competitive world where weeks ahead mean billions of dollars in revenue, speed matters. If a frontier model is reachable through an API, it can be distilled by querying it at scale and training on the transcripts. That single fact turned distillation from a lab technique into a competitive weapon.<br>This is why DeepSeek-R1 became such a flashpoint in early 2025.<br>Three Types of Distillation

Before we go further, it helps to know that “distillation” is not one technique and has expanded from Hinton’s initial concept.

Response-based distillation is the simplest and most commonly used distillation method today. The student trains on the teacher’s outputs. In the strong form, it matches the teacher’s full probability distribution over tokens, the soft labels Hinton originally described, which carry far more information than a single right answer because they encode how confident the teacher was and what it almost said instead. In the weak form, when all you have is an API that returns text, the student trains on the generated completions themselves.<br>Feature-based distillation goes deeper. Instead of matching outputs, the student matches the teacher’s internal representations, the activations in its hidden layers. While this transfers richer structure, it requires access to the teacher’s internals. You cannot do it through an API. And therefore is much more expensive and risky. It is a technique for labs distilling their own models, not for harvesting someone else’s.<br>Relation-based distillation is subtler still. It teaches the student to reproduce the relationships between the teacher’s representations, how examples sit relative to each other, rather than the representations themselves. It is more of a research direction than a workhorse, but it matters because it points at what distillation is really transferring: not answers, but structure.<br>The US/China capability war lives almost entirely in the first category. Response-based distillation is the only flavor you can run against a closed model you do not own, which is exactly why it works. And it has worked for China for almost 3 decades. While China established the socialist market economy already in 1978, the true opening to the world only happened in 2001 when China joined the World Trade Organization. If you wanted to invest in the growing Chinese market, foreign firms had to enter...