What Is Knowledge Distillation?
Knowledge distillation uses a teacher model's behaviour as a learning signal for a student. This chapter defines the idea, separates transfer from imitation, and introduces the first controlled experiment.
The first experiment in this book did not produce the easy result. A teacher network classified a small synthetic problem almost perfectly: about 99.9% mean held-out accuracy. A student with 228 parameters, against the teacher’s 17,412, was trained in two ways on the same data: once on the task labels alone, and once on the labels plus the teacher’s probabilities. Averaged over three seeds, the student that learned from the teacher was the less accurate of the two. Two of the three seeds favoured the label-only student; one favoured the student that learned from the teacher.
That outcome is not a reason to dismiss distillation. It is a reason to be precise about what distillation is, what it can carry from a teacher to a student, and how we would know whether anything useful arrived.
The motivating problem is ordinary. Suppose we have a model that performs a useful classification task well, but is too large or too expensive to run where we need it. We want a smaller model that performs the same task. The obvious approach is ordinary supervised learning: collect labelled examples, train the smaller model on those labels, and measure its accuracy on held-out data.
That approach may work. But it ignores part of what a trained teacher can supply. A trained teacher does not only produce a label. It produces a distribution over possible answers. For an input that is clearly one class, that distribution may be sharply peaked. For an ambiguous input, it may place substantial probability on more than one class. Those relative probabilities are a record of how the teacher behaves, and they can be used as an additional learning signal for a student.
That is the basic idea of knowledge distillation. Hinton et al. (2015) gave the modern formulation; an earlier model-compression approach had already trained a smaller model on labels produced by an ensemble (Buciluǎ et al., 2006). A teacher supplies behaviour; a student learns from that behaviour as well as from ground-truth labels. The student may end up smaller, more specialised, or cheaper to serve. It may also end up worse. The purpose of an experiment is to find out which.
This chapter defines the idea at an intuitive level. It explains what a teacher can add beyond a hard label, what is and is not transferred, and why a successful training run is not the same as a successful student. It then returns to that first controlled experiment, E1, to show what was measured, what was not, and why a negative accuracy result on one small synthetic task is a useful place to begin.
A smaller model learning from a larger one
Consider a four-class classification problem. An input belongs to one of four classes, and we have a labelled training set. We train a relatively large teacher model on that set. We then want a much smaller student model that can perform the same task.
There are at least two ways to train the student:
- Train it on the ground-truth labels only.
- Train it on the ground-truth labels and on the teacher’s predictions for the same inputs.
The first is the baseline. The second is distillation. The comparison is not between a good method and a bad method in the abstract. It is between two training procedures under otherwise comparable conditions. The student architecture, the data, the optimisation budget, and the evaluation set should be held as constant as the experiment allows.
The word teacher does not imply that the larger model is correct. It means that the larger model supplies a signal. The word student does not imply that the smaller model is a blank slate. It means that the smaller model is the one being trained. Both words describe roles in a training procedure, not guarantees about quality.
The diagram below shows the basic exchange. The student sees the same inputs as the teacher, but it receives two kinds of supervision: the task label and the teacher’s response. The final judgement is made on held-out task data, not on how closely the student imitates the teacher.
flowchart LR
X[Inputs] --> T[Teacher]
T --> R[Teacher response]
X --> S[Student]
Y[Ground-truth labels] --> S
R --> S
S --> P[Student predictions]
P --> E[Independent evaluation]
Figure 1. A teacher supplies an additional signal during student training. Evaluation remains independent of both the teacher and the training signal.
A teacher can be an ensemble, a larger single model, a model trained on more data, or a system that has been specialised for a domain. The book uses the general term because the mechanism does not depend on a particular architecture. What matters is that the teacher produces a response the student can learn from, and that the use of that response is permitted.
What the teacher adds beyond a label
A hard label says that an input belongs to class A. It does not say that class B was plausible, that class C was unlikely, or that two inputs labelled A are ambiguous in different ways. A teacher’s output can preserve some of that structure.
Consider an illustrative three-class example. The teacher produces the following probabilities for one input:
| Class | Teacher probability | Hard label |
|---|---|---|
| A | 0.70 | 1.00 |
| B | 0.25 | 0.00 |
| C | 0.05 | 0.00 |
The hard label records only that A is the selected class. The teacher distribution also records that B is a plausible alternative and C is much less plausible. For a student, that is a different kind of information. It says something about the boundary the teacher has learned, not only about the final decision. Hinton et al. (2015) describe this information in the relative probabilities of the incorrect answers and use a temperature to make it visible to the student.
Figure 2. The same illustrative distribution as the table above, drawn as bars. The hard label concentrates all mass on A; the teacher spreads mass across B and C as well. Illustrative values, not measured data.
The same example also shows a danger. The teacher may be wrong. A probability of 0.70 is not a guarantee that A is correct. The teacher may be overconfident, poorly calibrated, or trained on data that differs from the student’s deployment data. Teacher probabilities are evidence about teacher behaviour. They are not ground truth, and they are not a substitute for labels when labels are available.
The relative probabilities of the non-selected classes are sometimes called dark knowledge. The phrase is a metaphor. There is no separate substance called dark knowledge inside the model. There is information in the shape of the teacher’s distribution: which alternatives the teacher considers plausible, and how strongly. Whether that information helps a particular student is an empirical question.
The teacher may also produce logits rather than probabilities. A logit is the unnormalised score before the final softmax. Logits can preserve more of the teacher’s numerical behaviour, but they are not on the same scale as probabilities. Chapter 04 defines logits, softmax, temperature, and soft targets precisely; Chapter 05 derives the objective that uses them. For this chapter, it is enough to understand that a teacher response can be richer than a single label.
What is transferred?
Ordinary distillation does not copy the teacher’s parameters into the student. The student has its own architecture, its own weights, and often a different capacity. What is transferred is a pattern of behaviour, expressed through supervision on inputs.
That distinction matters. A student can learn to reproduce some of the teacher’s outputs without learning the same internal computation. It can also learn a useful approximation that differs from the teacher in detail but performs well on the task. Ba and Caruana (2014) showed that shallow networks can sometimes learn functions previously learned by deeper networks, which is one reason capacity is not the whole story. Neither outcome is guaranteed.
The output distribution is the most common signal, but it is not the only one. Later chapters examine methods that use:
- intermediate features, which ask the student to match internal representations;
- attention patterns, which transfer where a model looks;
- relations between examples, which transfer structure rather than individual values;
- whole sequences, which transfer behaviour over a longer output;
- tool-use trajectories and agent behaviour, which transfer a policy rather than a single prediction.
These are related forms of transfer, not identical mechanisms. The book treats model distillation, dataset distillation, reasoning distillation, and agent skill transfer as distinct families. Chapter 02 draws the boundaries. A method that uses synthetic data is not automatically a distillation method, and a method that trains on a teacher’s outputs is not automatically transferring a reasoning procedure.
What distillation can achieve
Distillation is attractive because it can address several different problems. The list below describes possible outcomes, not guarantees.
A smaller specialist model. A large general model can be used to train a smaller model for a narrower task. The student may be easier to deploy, but it may also lose robustness outside the task it was trained for.
Model compression. Distillation can reduce the number of parameters or the cost of inference. It is not the only compression method. Pruning, quantisation, low-rank approximation, and runtime techniques such as speculative decoding change different costs. A smaller checkpoint is not automatically a faster model; latency must be measured on the target hardware.
Lower serving cost. A student may be cheaper to run if it requires fewer parameters, less memory, or fewer floating-point operations. The actual saving depends on the deployment stack, batch size, sequence length, and hardware. A parameter count is a starting point, not a performance result.
Learning from an ensemble. Several models can be combined into a teacher signal, and a single student can be trained to approximate the ensemble’s behaviour. The student may capture some of the ensemble’s generalisation without running every member at inference time. It may also fail to capture the parts of the ensemble that matter.
Domain specialisation. A teacher with broad knowledge can supply supervision for a student that is optimised for a narrower domain. The risk is that the teacher’s behaviour outside the domain is irrelevant or misleading.
Instruction, reasoning, and tool-use transfer. A teacher can generate instruction-response pairs, reasoning traces, or tool-use trajectories for a student. These are active research areas. The evidence is uneven, the methods are not interchangeable, and the rights questions are often harder than the optimisation questions.
The common thread is that distillation changes the supervision available to a student. It does not by itself establish that the student is more accurate, better calibrated, cheaper, safer, or legally deployable. Each of those claims requires a different measurement.
Six ways distillation can fail
The first failure mode is the teacher. A teacher can be wrong, biased, or unrepresentative. A student trained to match the teacher can learn the teacher’s errors as well as its useful behaviour. The fact that a teacher performs well on average does not make every teacher response correct.
The second is capacity. A very large teacher may represent a function that a small student cannot approximate with the available architecture and training budget. The student may then underfit, or it may learn a simplified decision boundary that differs from both the teacher and the ground truth. Parameter count is only a proxy for capacity; architecture, optimisation, data, and the task itself also matter. Cho and Hariharan (2019) found that larger or more accurate teachers do not always produce better students, and Mirzadeh et al. (2020) explored intermediate teacher assistants as one response to the capacity gap. Chapter 03 treats this in detail.
The third is redundancy. If the teacher’s output adds little information beyond the ground-truth labels, distillation may increase training cost without improving the student. A teacher that is already near-perfect and highly confident may produce soft targets that are close to hard labels after temperature scaling. The additional signal may be small.
The fourth is a trade-off between accuracy and calibration. A student can become less confident, or better calibrated, without becoming more accurate. That may be useful in some applications and irrelevant in others. A single aggregate metric can hide the change. Chapter 05 and Chapter 34 separate correctness, teacher agreement, and calibration.
The fifth is cost. Distillation adds a teacher-forward pass, a second loss, and often a larger hyperparameter search. If the student does not improve, the extra training and inference cost is a real cost. A negative result is still a result, but it is not a free one.
Finally, the student may imitate the teacher without acquiring the capability the teacher was supposed to have. Agreement with the teacher is not the same as correctness on the task. A student that matches a wrong teacher perfectly is not a successful student.
The first experiment
E1 was designed to test the basic mechanism under controlled conditions. It used a four-class, two-dimensional spiral classification problem generated entirely inside the repository. The data has a non-linear decision boundary and known labels. Within each seed, every arm trained on the same partition; partitions were regenerated across seeds. No external dataset or pretrained model was used.
The teacher was a multilayer perceptron with two hidden layers of 128 units: 17,412 parameters. The student was a much smaller multilayer perceptron with one hidden layer of 32 units: 228 parameters. The teacher therefore had roughly 76 times as many parameters as the student.
The teacher was trained first with ordinary cross-entropy. Once trained, its parameters were frozen. The student was then trained under several supervision arms. The primary comparison was:
- Arm A, the baseline: the student was trained on ground-truth labels only.
- Arm B, mixed distillation: the student was trained on a mixture of the ground-truth cross-entropy and a loss that encouraged its temperature-scaled output distribution to match the teacher’s.
The mixed arm used a temperature of 2 and equal weight on the hard and soft
terms. The experiment used three training seeds and a held-out test set of 1,000
examples. The test set was not used for early stopping or hyperparameter
selection. The full protocol, configuration, and per-seed records are preserved
in experiments/E1-classical/; the primary group is
E1-20261112T233716Z-3bdf03, with provenance in planning/05-evidence.md. The
experiment ran on CPU with one PyTorch thread.
E1 also included a KL-only arm, a logit-MSE arm, temperature variants, and hard/soft weight variants. Those variants are reported in the experiment record and support later chapters. Chapter 01 uses only the primary comparison; it does not need the full matrix to make its point.
What E1 found
Two measurements need a word of explanation before the results. Held-out accuracy is the task measure: the fraction of the 1,000 test examples that a model classified correctly. Expected calibration error (ECE) measures something different: whether a model’s confidence matches how often it is right. It is the weighted mean absolute gap between confidence and accuracy across 15 equal-width confidence bins, using the maximum class probability as confidence; Chapter 34 develops the measurement. A model can be accurate and poorly calibrated, or inaccurate and well calibrated.
The teacher was very accurate on this task: approximately 99.9% mean held-out accuracy across the three seeds. Its mean expected calibration error was about 0.006, so its confidence was close to its measured accuracy. The student was much smaller and much less accurate. The measured means were:
| Quantity | Teacher | Student A: hard labels | Student B: mixed distillation |
|---|---|---|---|
| Parameters | 17,412 | 228 | 228 |
| Held-out accuracy | 99.9% | 66.87% | 64.20% |
| Cross-entropy | — | 0.8733 | 0.8651 |
| Expected calibration error | — | 0.1461 | 0.0894 |
| Teacher agreement | — | 66.97% | 64.30% |
Read the table one row at a time, and keep the rows apart. Accuracy says how often each student was right. Cross-entropy and ECE describe the quality of its probabilities. Teacher agreement says how often it chose the teacher’s answer, which is a measure of imitation, not of correctness. None of the rows says why the two students differ.
The distilled student was worse on mean accuracy by 2.67 percentage points. Two of the three seeds favoured the hard-label baseline; one favoured the distilled student. The mean difference is negative, and the seed-level variation is large enough that the result should not be treated as a precise effect size.
The pattern of secondary metrics is different. Arm B had slightly lower cross-entropy and substantially lower expected calibration error. In this run, the student trained with the teacher signal produced probability estimates that were better aligned with its own correctness, even though its top-1 accuracy was lower. That is an accuracy/calibration trade-off, not a demonstration that distillation improved the student’s capability.
The teacher-agreement numbers are also instructive. Agreement was about 66.97% for the baseline and 64.30% for the mixed student. Because the teacher was near-perfect on this task, the two aggregates were almost identical on this test set. That is a property of this experiment, not a general law. A student can agree with a teacher while being wrong, and a student can disagree with a teacher while being right. Agreement is a diagnostic; held-out correctness is the task measure.
The experiment does not explain why the distilled student was less accurate. Several explanations are consistent with the data:
- the student may lack the capacity to approximate the teacher’s function;
- the teacher was extremely confident, so the soft targets may have added little information after temperature scaling;
- the synthetic task and the small student may favour the simpler baseline;
- the bounded hyperparameter search may not have found a better configuration.
None of these explanations was isolated experimentally. They are hypotheses for later work, not findings of E1. The result does not show that distillation generally fails, and it does not show that it generally helps. It shows that the mechanism must be measured, and that a strong teacher is not sufficient evidence for a strong student.
The limitations are part of the result:
- one synthetic dataset;
- three seeds;
- very small teacher and student models;
- a near-perfect teacher;
- a bounded temperature and weight search;
- host-sensitive latency measurements;
- no causal isolation of the accuracy deficit;
- no external replication.
The latency measurements deserve a specific caution. A smaller model can have fewer parameters and still fail to reduce latency on a particular machine. In E1, the latency measurements were sensitive to host scheduling, and a post-run check on saved checkpoints produced different values from the in-run measurements. The book therefore treats parameter count and artefact size as the stable cost measures from E1, and treats latency as a diagnostic that must be remeasured on the target deployment environment.
Why permission comes before training
A model can be technically accessible and still not be available for the intended use. A public checkpoint may be downloadable without permission to train on its outputs or redistribute a derivative. An API may return useful text while its terms restrict using that text to train a competing model. A dataset may be viewable without a licence that permits training. These are separate questions from whether the optimisation works.
E1 avoided these complications deliberately. Its data, teacher, and student were created inside the repository. The experiment has a documented rights record for the specific action it performs: training and evaluating the E1 models. That record is not a general permission to use other models or data, and it does not make the resulting student automatically redistributable.
For a real project, the questions arrive in a different order from the code. Can we use the teacher? Can we use its outputs? Can we use the training data? Can we distribute the student? Does commercial use change the answer? The book treats these as engineering constraints, not as an appendix added after the model is trained. Chapters 36–39 develop the framework. Chapter 01 only establishes the principle: technical access is not the same as permission.
Limits of this chapter’s evidence
The E1 numbers in this chapter are book results: measurements from the frozen protocol and manifests recorded with the experiment. They are means across three seeds on one synthetic task. They are not results reported by Hinton et al., Buciluǎ et al., or any other external source, and they do not establish a general effect.
The external sources in this chapter are used for the mechanism and its history: the idea of training a smaller model on a teacher’s outputs, the use of soft targets and temperature, and the observation that teacher quality alone does not determine student quality. Detailed derivations and implementation follow in Chapters 04–06.
What you should now be able to do
After this chapter you should be able to:
- explain, without equations, what a teacher’s distribution can carry that a hard label cannot, and why that information may or may not help a particular student;
- describe teacher and student as roles in a training procedure, without assuming that the teacher is correct;
- keep three measurements apart: teacher quality, student correctness on held-out data, and agreement between teacher and student;
- read a result like E1’s without turning a calibration change into a claim about capability;
- name the permission questions that come before training on a teacher or its outputs.
What comes next
The rest of the book builds the machinery that this chapter has kept at arm’s length.
Chapter 02 separates model distillation from dataset distillation, reasoning distillation, agent skill transfer, fine-tuning, self-training, compression, and other forms of AI improvement. Chapter 03 defines the teacher and the student, including the access regimes and capacity constraints that shape what can be transferred. Chapter 04 develops the probability and numerical foundations: logits, softmax, temperature, and soft targets. Chapter 05 derives the distillation objective and turns the intuitive comparison in this chapter into a testable loss.
Part II extends the mechanism to response and logit matching, black-box access, feature and relational supervision, sequence-level distillation, and multiple teachers. Part III treats data as an engineered artefact. Part IV moves to language models, reasoning, verification, on-policy learning, self-distillation, preference data, and continual adaptation. Part V applies the ideas to retrieval, code, agents, memory, multimodal systems, and generative media. Part VI measures what actually transferred. Part VII examines the legal conditions. Part VIII integrates the workbench and reports the controlled experiments.
The thread running through all of it is the distinction introduced here:
Distillation is a mechanism for transferring behaviour. Whether it transfers useful capability is an experimental question.
E1 is the first experiment in that inquiry. It gives us a working teacher–student system, a measured comparison, and a result that resists the easy story. The student did not become more accurate by learning from a much stronger teacher. It did produce better-calibrated probabilities. Both observations matter, and neither one is enough to justify a general claim.
The next chapter asks a more basic question: if so many AI methods change a model’s behaviour, what makes distillation a distinct method at all?
References
Ba, J. and Caruana, R. (2014). Do Deep Nets Really Need to be Deep? arXiv:1312.6184. https://arxiv.org/abs/1312.6184
Buciluǎ, C., Caruana, R. and Niculescu-Mizil, A. (2006). Model Compression. Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 535–541. https://doi.org/10.1145/1150402.1150464
Cho, J. H. and Hariharan, B. (2019). On the Efficacy of Knowledge Distillation. Proceedings of the IEEE/CVF International Conference on Computer Vision. arXiv:1910.01348. https://arxiv.org/abs/1910.01348
Hinton, G., Vinyals, O. and Dean, J. (2015). Distilling the Knowledge in a Neural Network. NIPS 2014 Deep Learning Workshop. arXiv:1503.02531. https://arxiv.org/abs/1503.02531
Mirzadeh, S.-I., Farajtabar, M., Li, A., Levine, N., Matsukawa, A. and Ghasemzadeh, H. (2020). Improved Knowledge Distillation via Teacher Assistant. Proceedings of the AAAI Conference on Artificial Intelligence. arXiv:1902.03393. https://arxiv.org/abs/1902.03393