The Teacher and the Student
A distillation pair is viable only when the teacher's signal, the student's capacity, and the interface between them are compatible. This chapter defines the roles, the access regime, capacity, and the E1 pair.
Two models can each be excellent and still be useless as a pair. If token id 42
is "cat" in the teacher’s vocabulary and "the" in the student’s, a loss that
matches teacher logit 42 to student logit 42 compares different units, however
good either model is. If the teacher is reachable only through a service whose
terms restrict using its outputs to train a competing model, its accuracy does
not settle whether it can be used at all.
A distillation system has two roles. The teacher supplies supervision, and the student is trained to acquire the target behaviour. The words sound simple, but a distillation project can fail at the pair rather than at the loss. The teacher may be accurate but inaccessible; the student may be too small for the teacher’s function; the output spaces may not line up; the tokenisers may assign different meanings to the same index; or the teacher may be technically available but not permitted for the intended use.
This chapter moves from the taxonomy of Chapter 02 to the selection of a workable pair. It defines four concepts that the rest of the book uses: the teacher, the student, the access regime, and model capacity. It then explains why two individually capable models can still be a poor pair, why a larger teacher is not automatically a better teacher, and how the capacity gap can be described without pretending that it has been proved. E1 supplies a concrete pair and a concrete warning.
The central question is:
Given a task, a teacher, and a student, what can the student actually access, and what determines whether knowledge transfer is feasible?
The short answer is compatibility. A viable pair needs a teaching signal that the student can use, architectures and output spaces that can be connected, a student with enough effective capacity for the target behaviour, a training budget that makes the comparison meaningful, and permission for every source and artefact involved.
A teacher is more than a larger model
The teacher is a model or system that supplies supervision to a learner. It may be a larger neural network, but it need not be. It can be a model with a different architecture, a specialist model, an ensemble, a verifier or expert process, an earlier checkpoint, or a frozen or differently conditioned version of the student.
The teacher role carries several properties that must be recorded separately:
- identity and version, because a checkpoint or provider model can change;
- architecture and parameter count, because these shape what the teacher can represent and what the student may need to match;
- task accuracy, because a teacher that cannot perform the task supplies poor supervision;
- calibration and error structure, because confidence is not correctness;
- access regime, because the student cannot learn from information the teacher does not expose;
- rights, because technical access is not permission.
Teacher quality is not the same as teacher accuracy. A teacher can be accurate on average and badly wrong on the examples that matter to the student. It can be overconfident, poorly calibrated, or trained on a different distribution. A teacher can also be an ensemble whose members disagree; the disagreement may be useful information or a source of noise. None of these properties is captured by a single accuracy number.
The role is also not a rank. A student can be larger than its teacher, and a teacher can be a frozen copy of the student from an earlier training stage. The words describe who supplies the signal and who is being trained, not who is better. An ensemble can supply a combined signal, as in the classical distillation setup (Hinton et al., 2015).
The student is a model with its own constraints
The student is the model being trained to acquire the target behaviour. Its specification includes the architecture, initialisation, parameter count, training data, learning objective, available compute, and intended deployment environment. A student may be a small specialist, a general-purpose model, a compressed copy of a larger system, or a model with a different architecture from the teacher.
The student’s capacity is not a property of its parameter count alone. A student with fewer parameters can sometimes learn a restricted task very well, while a larger student can fail because its architecture, optimisation, or data are unsuitable. A student can also match or exceed a teacher on a narrow benchmark without possessing the teacher’s general capabilities. That is not a paradox; it is a reminder that the evaluation task defines the claim.
Student quality is measured independently on held-out task data. Teacher agreement is a diagnostic, not a substitute for correctness. A student that matches a wrong teacher is not a successful student, and a student that disagrees with a correct teacher may still be right.
What the student can access
The access regime is the information a teacher makes available to the learning process. It is a property of the interface, not of the teacher’s marketing category. The practical ladder runs from full internal access to top-1 labels:
| Level | Typical signal | What it enables | Main limitation |
|---|---|---|---|
| 1 | Weights and internal representations | Full model inspection; feature and weight-based methods | Requires local or privileged access |
| 2 | Gradients and intermediate activations | Gradient-based transfer and feature matching | Not usually exposed by services |
| 3 | Full logits | Logit matching, temperature scaling, probability construction | Requires output-space compatibility |
| 4 | Full probability distributions | Response-based distillation and calibration analysis | Normalisation and truncation can hide information |
| 5 | Top-k probabilities or scores | Partial distribution matching | Missing mass and ranking truncation |
| 6 | Generated sequences or sampled responses | Sequence-level and response-based supervision | Sampling variance and query cost |
| 7 | Top-1 labels only | Hard-label distillation or ordinary supervised training | No soft information or internal state |
The ladder is a useful progression of information access, not a universal hierarchy. A model may expose features but not gradients; a service may return scores without full logits; an API may return text without token-level probabilities at all. The access regime should record the signals that are actually available, not the label that the provider uses.
The three conventional names are still useful:
- White-box access exposes weights, and usually internal activations or gradients. It supports the widest range of objectives.
- Grey-box access exposes some internal signal, such as logits, probabilities, top-k scores, or features, without full weights.
- Black-box access exposes only sampled outputs or top-1 labels. It supports response-based or hard-label supervision, with higher query and variance costs.
Access determines which objectives are feasible. Full logits permit logit matching; full probabilities permit response-based distillation; top-1 labels permit hard-label training but not a distribution match. Access also determines cost: a black-box teacher may require many queries, a rate limit, or a budget. Chapter 07 treats black-box access in detail.
Access is not permission. A model can expose logits through an API and still be subject to terms that prohibit using those outputs to train a competing model. The access regime is an engineering record; the rights gate is a separate decision.
Capacity is not a parameter count
Model capacity is the effective ability of a model to represent and generalise the target behaviour. Parameter count is a proxy, not a definition. Two models with the same number of parameters can differ in width, depth, representation quality, inductive bias, training data, optimisation, task difficulty, available compute, and specialisation. A parameter count also says nothing about whether the model was trained well or whether its representations are useful for the student’s task.
A simple descriptive ratio is
It is useful to separate three sources of student error:
E_S ≈ E_approx(F_S, f_T) + E_est(D, F_S) + E_opt(A, F_S)
In words, a student can fall short of its teacher for three different reasons: no function it can represent matches the teacher closely; it saw too little data to identify a good one; or training did not reach the best one it could have represented. The terms are defined precisely below.
Here E_S is the student’s expected error relative to the teacher’s function
f_T, F_S is the function class the student can represent, D is the training
data, and A is the optimisation procedure.
E_approx is the best error any function in F_S can achieve against f_T;
E_est reflects finite data; and E_opt reflects the gap between the best
function in the class and the one the optimiser finds. This is a bookkeeping
device, not a theorem about a particular model. The terms interact, and a change
in architecture can affect all three. Task error against ground truth additionally
includes the teacher’s own error; that term is not in this decomposition.
A small student may have high approximation error for a complex teacher function. A larger student may reduce that term but increase estimation or optimisation difficulty. The same trade-off is often described in terms of bias and variance: a simpler student has higher bias, a more flexible student can have higher variance, and the balance depends on the data, task, and optimiser. The useful question is not “how many parameters does the student have?” but “can this student represent the target behaviour, and can it learn that behaviour from the available signal and data?” Ba and Caruana (2014) showed that shallow networks can sometimes learn functions previously learned by deeper networks; depth is not the only determinant of useful capacity.
An illustrative example makes the point without measuring anything. Suppose the
teacher’s function is f_T(x) = sin(3πx) on [0, 1], and the student is a linear
function f_S(x) = ax + b. No choice of a and b matches the teacher
everywhere. A higher-capacity student can reduce the approximation error, but it
may also require more data or more careful optimisation. The example is a
conceptual illustration, not a model of E1.
The capacity gap
The capacity gap is the mismatch between the complexity of the teacher’s behaviour and the student’s ability to represent and learn it under the available conditions. The gap is not simply a parameter ratio. It can come from depth, width, representation quality, task difficulty, data, optimisation, or the teacher’s own function.
Cho and Hariharan (2019) report that larger or more accurate teachers often did not make good teachers for small students in their experiments. They attribute this to mismatched capacity, report that small students can fail to mimic large teachers, and report that a simple sequence of distillation steps was ineffective in their setting. They also report that stopping the teacher’s training early can mitigate the effect. Those are experimental observations in their settings, not a universal law.
Mirzadeh et al. (2020) also report that student performance degrades when the gap between teacher and student is large. They introduce an intermediate teacher assistant and report improved results across their tasks. The proposed mechanism is that an intermediate model bridges the gap between a large teacher and a small student. The approach adds a model, a training stage, and additional cost; it is not a free fix.
The two papers are a useful warning against a single story. Cho and Hariharan report that simple multi-step distillation was ineffective in their experiments; Mirzadeh et al. report gains from a teacher assistant in theirs. The difference may depend on the task, architectures, training procedure, and how the assistant is produced. That is a plausible explanation, not a result established here.
E1 is consistent with a capacity gap: its teacher has 17,412 parameters and its student has 228, and the distilled student did not improve on the hard-label baseline’s mean held-out accuracy. But E1 does not isolate capacity as the cause. Optimisation, teacher confidence, the training signal, and the synthetic task are also plausible explanations. The chapter treats the capacity gap as a diagnostic hypothesis, not as a proven explanation of E1.
| Role | Model | Mean held-out accuracy |
|---|---|---|
| Teacher | MLP 2 -> 128 -> 128 -> 4, 17,412 parameters |
99.9% |
| Baseline student | MLP 2 -> 32 -> 4, 228 parameters, hard labels (Arm A) |
66.87% |
| Distilled student | Same 228-parameter student, mixed KD at \(T=2\), \(\lambda=0.5\) (Arm B) | 64.20% |
The table records the E1 pair and its two student outcomes on the same held-out test set. It does not establish that the 76-fold parameter difference caused the accuracy difference.
A teacher assistant is one response to the gap:
flowchart LR
T[Teacher] --> A[Intermediate-capacity assistant]
A --> S[Small student]
Figure 1. A teacher assistant bridges the gap in two stages: the assistant learns from the teacher, then the student learns from the assistant.
The assistant is trained from the teacher, then the student is trained from the assistant. The hypothesis is that the intermediate signal is easier for the student to use. The cost is an extra model, an extra training stage, and another artefact to evaluate and govern. The approach should be tested against the direct teacher-student baseline, with the same student, data, and evaluation protocol.
Teacher–student compatibility
Two capable models can still form a poor pair. Compatibility has several dimensions, and a failure in any one can invalidate a training run.
Output-space compatibility
The teacher and student must represent the same target labels or outputs, or there must be an explicit mapping between them. A teacher with ten classes and a student with four cannot match logits index by index without deciding what the extra classes mean. A teacher trained on a different label set may produce labels the student cannot represent. The E1 pair is compatible: both models take the same two-dimensional input and produce four class logits.
Tokeniser compatibility
Language-model teachers and students may use different tokenisers and
vocabularies. Suppose token id 42 is "cat" in the teacher’s vocabulary and
"the" in the student’s. A loss that matches teacher logit 42 to student logit 42
compares different units. Even two tokenisers with the same vocabulary size can
assign different meanings to the same index.
Token-level logit matching is therefore invalid without a shared token space or an explicit alignment. Sequence-level supervision is one alternative: the student learns from teacher-generated sequences rather than from index-aligned logits; Kim and Rush (2016) introduce this sequence-level formulation for neural machine translation. Applying it under tokeniser mismatch is a book inference, not a claim made in that paper. An alignment or projection is another possibility, but it is a design choice that must be validated, not assumed. Chapter 10 treats sequence-level methods, and Chapter 17 treats transformer-specific questions.
Representation compatibility
A teacher and student may have different hidden dimensions, layer counts, or
architectures. Feature matching then requires a projection or an alignment
mechanism. FitNets, for example, uses hint and guided layers to connect a thin
student to a teacher’s intermediate representations (Romero et al., 2015). The
projection adds parameters and introduces a design choice. A dimension mismatch
is not solved by assuming that the first d features correspond to the first
d features of the other model.
Feature access also depends on the access regime. A model may expose weights but not activations, or activations but not gradients. Chapter 08 treats the objectives and their assumptions.
Task compatibility
The teacher may solve a broader task than the student needs. A specialist student can be trained on a narrower input distribution, but the teacher’s behaviour outside that distribution may be irrelevant or misleading. The pair should have a defined target task, a defined evaluation slice, and a rule for handling inputs outside the slice. A student that performs well on the narrow task does not thereby acquire the teacher’s general capability.
Interface compatibility
The interface determines what can be queried and at what cost. A local model can be inspected and run without a network service; a hosted API may expose only sampled text; a restricted service may impose rate limits, output truncation, or a query budget. These are engineering constraints, not implementation details that can be discovered after training begins. Chapter 07 treats black-box access and query budgets in depth.
E1 as a concrete pair
E1 provides a recorded teacher-student pair with a clean rights position. The
teacher is a multilayer perceptron with layers 2 -> 128 -> 128 -> 4, 17,412
parameters, trained on the four-class spiral. The student is a multilayer
perceptron with layers 2 -> 32 -> 4, 228 parameters. The pair shares the input
space, the four-class output space, the task, the training and validation splits,
and the held-out test set.
| Property | Teacher | Student |
|---|---|---|
| Architecture | MLP 2 -> 128 -> 128 -> 4 |
MLP 2 -> 32 -> 4 |
| Parameters | 17,412 | 228 |
| Input space | Two continuous features | Two continuous features |
| Output space | Four classes | Four classes |
| Intended task | Four-class spiral | Four-class spiral |
| Available supervision | Weights, activations, logits, probabilities | Labels plus teacher logits and probabilities |
| Rights record | RM-101 |
RM-101 |
The teacher was trained first with ordinary cross-entropy, then frozen. The student was trained under a hard-label baseline and a mixed-distillation arm. The student architecture, initialisation, data order, optimiser, batch size, and maximum epochs were held constant across arms. The primary group used three seeds and a held-out test set of 1,000 examples. The test set was not used for early stopping or hyperparameter selection.
The teacher reached approximately 99.9% mean held-out accuracy. The hard-label baseline reached 66.87% mean accuracy; the mixed-distillation student reached 64.20%. The mixed student had lower cross-entropy and lower measured calibration error. Chapter 01 reports the full comparison; this chapter uses the pair, not the result table.
The pair was chosen to make the compression question concrete. The teacher is large enough to learn the spiral almost perfectly, and the student is small enough that the question “can this behaviour be transferred?” is meaningful. The shared input and output spaces remove avoidable interface problems, and the self-created data and models avoid an external rights dependency. These are design reasons, not a claim that the pair is representative of every distillation problem.
The 76-fold parameter difference is descriptive. It does not prove that capacity caused the accuracy difference. E1’s own limitations list optimisation, teacher confidence, the training signal, the synthetic task, and the bounded hyperparameter search as alternatives. The correct conclusion is that the pair was compatible and the transfer was measured, not that the pair failed for a single known reason.
Three compatibility cases
A — Compatible classifier pair
E1 is the compatible case. The teacher and student share the input space, the
four-class output space, the task, the splits, and the evaluation protocol. The
teacher’s logits and probabilities are available; the student can use them. The
teacher and student are both self-created, and RM-101 covers the action
train_and_evaluate_e1. The baseline and distilled student use the same student
architecture, initialisation, data order, and budget.
The compatibility check is not “are both models good?” It is “can the student use the signal the teacher supplies, on the same task, under the same evaluation conditions?” For E1, the answer is yes. The result is a separate question.
B — Incompatible language-model pair
Consider an illustrative teacher with a 50,000-token vocabulary and a student with a 32,000-token vocabulary. The vocabularies are not a shared coordinate system. A teacher logit at index 42 and a student logit at index 42 may refer to different strings, and the teacher may have tokens the student cannot produce. Direct token-level logit matching is invalid.
A sequence-level approach avoids the index alignment problem: the student is trained on teacher-generated sequences, with the sequence as the target (Kim and Rush, 2016, for the sequence-level formulation; the tokeniser-mismatch application is a book inference). An explicit alignment or projection is another option, but it must be designed and evaluated; this chapter does not claim that an unimplemented alignment algorithm has been verified. The safest first check is to record both tokenisers, the vocabulary sizes, and the special-token conventions before choosing an objective.
C — Restricted black-box teacher
Consider a local mock teacher that returns only its highest-ranked class for each query. The available signal is a top-1 label. The missing information includes probabilities, logits, features, and gradients. Hard-label training or hard-label distillation remains possible: the student can learn from the teacher’s labels, with or without ground-truth labels.
The constraints are query cost, latency, rate limits, and coverage. A query budget may make a large teaching set expensive; a rate limit may make training slow; a top-1 label may hide ambiguity that a probability distribution would expose. The rights questions are separate: the teacher’s terms may restrict using its outputs to train another model, and the student’s distribution may have additional conditions. A non-empty model ID does not answer any of these questions.
Specifying the pair reproducibly
Chapter 03 adds a small record layer at examples/common/specs.py. It defines
AccessRegime, TeacherSpec, and StudentSpec as frozen dataclasses, plus
compatibility_issues and assert_compatible. The records carry model identity,
version, architecture, parameter count, input and output spaces, tokeniser where
relevant, intended task, access signals, and a rights-record identifier.
The E1 pair can be recorded as follows:
from examples.common.specs import (
AccessRegime,
StudentSpec,
TeacherSpec,
assert_compatible,
)
teacher = TeacherSpec(
model_id="e1-mlp-teacher",
version="protocol-1.1",
architecture="MLP 2-128-128-4",
parameter_count=17_412,
input_space="features:2",
output_space="classes:4",
intended_task="four_class_spiral",
rights_record="RM-101",
access=AccessRegime(
weights=True,
activations=True,
logits=True,
probabilities=True,
),
)
student = StudentSpec(
model_id="e1-mlp-student",
version="protocol-1.1",
architecture="MLP 2-32-4",
parameter_count=228,
input_space="features:2",
output_space="classes:4",
intended_task="four_class_spiral",
rights_record="RM-101",
)
assert_compatible(teacher, student)
The access record describes capabilities, not a provider label. AccessRegime
rejects an empty declaration, gradients without weights, labels_only combined
with other signals, and invalid query or top-k values. The kind property derives
white-box, grey-box, or black-box from the recorded signals.
compatibility_issues checks input space, output space, task, and tokeniser.
assert_compatible raises SpecError when the pair cannot share the recorded
supervision. The records serialise to stable JSON and round-trip through
from_dict. Focused tests cover valid records, missing fields, unknown fields,
incompatible output spaces, tokeniser mismatches, invalid access declarations,
serialisation, the E1 parameter counts, and the association with RM-101 through
the existing rights gate.
The E1 training code is unchanged. The specification describes the recorded pair; it does not retrain the models or alter the frozen evaluation. A specification is a record of identity and interface, not a licence. The rights gate remains authoritative.
Rights and deployment constraints
E1 avoided external rights dependencies by creating its data, teacher, and
student inside the repository. RM-101 covers the action
train_and_evaluate_e1; it does not authorise redistribution of the student or
use of the model for a different action.
For an external teacher, five questions are separate:
- Technical access: can the student obtain the signal?
- Contractual permission: do the model licence or service terms permit the intended training action?
- Output rights: may the teacher’s outputs be used as training data?
- Data rights: may the underlying training data be used, and does it contain personal or confidential information?
- Distribution: may the resulting student be published, commercialised, or redistributed?
A model may be downloadable without permission to distil it. The Gemma Terms of Use, for example, treat distillation using intermediate representations or synthetic outputs as a Model Derivative, with distribution conditions and use restrictions (L-004; retrieved 12 November 2026). A commercial API may return useful outputs while its terms restrict using them to train a competing model (L-016; retrieved 12 November 2026). The specification records the model ID and the access signals; it does not assert permission.
A teacher/student record with a non-empty rights_record is not thereby
authorised. The record identifies the decision that the existing gate must check:
the source, the action, the status, and the review date. Unknown is not
permitted. Chapters 36–39 develop the legal framework.
Limits of this chapter’s evidence
E1 is one recorded pair on one synthetic task, measured with three seeds. Its 76-fold parameter ratio is descriptive, and the chapter treats the capacity gap as a hypothesis for E1, not as an explanation of its result. The capacity-gap and teacher-assistant findings belong to the experimental settings of Cho and Hariharan (2019) and Mirzadeh et al. (2020); this chapter does not reproduce them. The language-model and black-box cases are illustrations, not measurements, and the use of sequence-level supervision under tokeniser mismatch is a book inference. The error decomposition is a bookkeeping device. The specification layer is tested as a record format: passing its compatibility check does not show that a transfer will work, and a rights-record identifier is not permission.
What comes next
The pair is now specified. The next question is what the teacher’s outputs mean. Chapter 04 defines logits, probabilities, softmax, temperature, and soft targets. It explains how a distribution can carry information beyond a hard label and how temperature changes that distribution. Chapter 05 then derives the distillation objective and connects it to the E1 protocol.
The sequence is deliberate. A pair is not viable because the teacher is large or the student is small. It is viable when the student can access a useful signal, the spaces and task are compatible, the student has enough effective capacity, the training budget is adequate, and the rights record covers the action. Those conditions are the subject of this chapter; the mathematics of the signal is the subject of the next.
References
Ba, J. and Caruana, R. (2014). Do Deep Nets Really Need to be Deep? arXiv:1312.6184. https://arxiv.org/abs/1312.6184
Cho, J. H. and Hariharan, B. (2019). On the Efficacy of Knowledge Distillation. Proceedings of the IEEE/CVF International Conference on Computer Vision. arXiv:1910.01348. https://arxiv.org/abs/1910.01348
Hinton, G., Vinyals, O. and Dean, J. (2015). Distilling the Knowledge in a Neural Network. NIPS 2014 Deep Learning Workshop. arXiv:1503.02531. https://arxiv.org/abs/1503.02531
Kim, Y. and Rush, A. M. (2016). Sequence-Level Knowledge Distillation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. arXiv:1606.07947. https://arxiv.org/abs/1606.07947
Mirzadeh, S.-I., Farajtabar, M., Li, A., Levine, N., Matsukawa, A. and Ghasemzadeh, H. (2020). Improved Knowledge Distillation via Teacher Assistant. Proceedings of the AAAI Conference on Artificial Intelligence. arXiv:1902.03393. https://arxiv.org/abs/1902.03393
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C. and Bengio, Y. (2015). FitNets: Hints for Thin Deep Nets. International Conference on Learning Representations. arXiv:1412.6550. https://arxiv.org/abs/1412.6550
Legal sources
Gemma Terms of Use. Last modified 1 April 2026. https://ai.google.dev/gemma/terms (retrieved 12 November 2026).
OpenAI Services Agreement. Updated 1 December 2025; effective 1 January 2026. https://openai.com/policies/services-agreement/ (retrieved 12 November 2026).