Model Distillation

Model distillation (also called knowledge distillation) is a machine learning technique in which a smaller “student” model is trained to replicate the behavior and reasoning of a larger, pre-trained “teacher” model. By learning from the teacher’s detailed outputs rather than simple right-or-wrong labels, the student captures much of the teacher’s capabilities while becoming significantly faster, smaller, and cheaper to run.

The Teacher-Student Analogy

Think of it like a master chef teaching an apprentice. The master chef (the teacher model) has years of experience and knows how to balance complex flavors. Instead of the apprentice (the student model) spending years learning from scratch, they watch the master work, taste the results, and learn the techniques used to achieve them. The apprentice doesn’t need to be as experienced as the master—they just need to capture enough of the master’s judgment to produce a meal that tastes nearly as good.

How Distillation Works

In traditional training, a model learns from “ground-truth” labels—essentially being told whether an answer is right or wrong. In distillation, the student goes further: it trains on the teacher’s soft targets, which are probability distributions that reveal how confident the teacher is about each possible answer.

For example, if the teacher model is classifying an image of a golden retriever, its soft target might say “85% golden retriever, 10% labrador, 3% beagle, 2% other.” A hard label would only say “golden retriever.” The soft target gives the student a richer map of the teacher’s reasoning—showing which answers the teacher considered close, and how much weight it gave each possibility.

In the context of large language models, distillation has expanded beyond output probabilities to include matching the teacher’s reasoning steps, intermediate representations, and chain-of-thought patterns. This lets the student model learn not just what the teacher concluded, but how it got there.

Common approaches include:

• Logit-based distillation: Matching the teacher’s final output probability distribution.
• Feature-based distillation: Matching the teacher’s internal layer representations.
• Reasoning distillation: Training the student to reproduce the teacher’s step-by-step reasoning process.
• Self-distillation: A model distills knowledge into a smaller version of itself.

Legitimate Technique, Contested Applications

Distillation is a standard, widely accepted practice in deep learning. It is how companies deploy powerful AI capabilities on phones, laptops, and edge devices without needing massive server farms. When a company trains a small model using its own large model’s outputs, distillation is simply an optimization step—no different in principle from any other model compression method.

The technique enters contested territory when the teacher model belongs to someone else. Because distillation can transfer high-level capabilities—including reasoning and tool use—from a proprietary model into a smaller, independently distributed model, it raises questions about intellectual property, terms of service, and the effectiveness of traditional controls. If a smaller distilled model retains most of a larger model’s capabilities, restrictions on the larger model’s hardware or deployment may not prevent those capabilities from spreading.

This tension has made distillation a focal point in AI policy debates. The core question is not whether distillation itself is valid—it is—but how to govern the extraction and transfer of capabilities when the source model is proprietary and the access methods may violate the provider’s terms.

Key Takeaways

• Model distillation trains a smaller student model to replicate a larger teacher model’s behavior, capturing most of its capabilities at a fraction of the computational cost.
• The student learns from the teacher’s nuanced probability outputs (“soft targets”), not just final answers—giving it a richer understanding of the teacher’s reasoning.
• Distillation is a legitimate and widely used optimization technique, central to deploying AI on resource-constrained hardware.
• When applied to proprietary models without authorization, distillation raises intellectual property and policy questions—particularly around the effectiveness of hardware export controls and terms-of-service enforcement.