← Distillation From First Principles

Classification, Embeddings, and Retrieval

Can a compact bi-encoder preserve a cross-encoder teacher's ranking behaviour while retaining independently measured retrieval relevance? Contrastive objectives, score distillation, hard negatives, Recall@k, MRR, nDCG, and a bounded E2 study with a weak, seed-concentrated result.

A classifier chooses one label from a fixed set. A retriever ranks a corpus whose size is not fixed, for a query it has never seen. The difference sounds organisational until the teacher arrives: a cross-encoder can read a query and a document together and score the pair, but it must do that for every document at query time. A bi-encoder must turn each side into a vector once, then compare vectors. The compact student is therefore not merely smaller; it is a different kind of function:

Can a compact student preserve a larger teacher’s ranking behaviour while retaining independently measured retrieval relevance?

On our bounded study, the answer is a qualified and uncomfortable “partly”. Adding teacher scores to a contrastive student raised mean nDCG@5 from 0.502 to 0.529, but one seed supplied the entire gain, one seed tied, and one was marginally worse. The teacher-only arm was clearly worse than both. The teacher itself ranked far better than any student (nDCG@5 0.856–0.914), and the best arm’s teacher–student top-1 agreement stayed near 0.27. The student learned some ranking; it did not learn the teacher’s ranking. Those are different claims, and the experiment exists to keep them apart.

Retrieval is ranking, not classification

A retrieval system receives a query \(q\), a corpus \(\mathcal{D}=\{d_1,\ldots,d_N\}\), and a scorer \(s(q,d)\). It returns the documents ordered by \(s\), usually truncated at \(k\). The task’s relevance judgements are a separate object: for each query, a set \(\mathcal{R}_q \subseteq \mathcal{D}\) of documents that actually answer it. In a classified task the label set is fixed and the model chooses one; in retrieval the candidate set is the corpus, and the model chooses an ordering. A classifier’s accuracy is therefore not a retrieval metric, and a retriever’s top-1 score is not its whole ranking.

Our corpus makes the distinction concrete. There are 16 entities and six relations, giving 96 \((entity, relation)\) pairs. Each pair has exactly one passage that states the relation for that entity, and one query that asks for it. A query is the token sequence [QM, ENT03, WROTE]; its relevant passage begins [ENT03, WROTE, V18, ...] and continues with five filler tokens. A passage that begins [ENT03, FOUNDIN, ...] shares the entity but violates the relation; a passage that begins [ENT07, WROTE, ...] shares the relation but names the wrong entity. Both are plausible distractors, and both are non-relevant by construction. Relevance is a fact recorded when the corpus is generated, not a model’s opinion.

That last sentence carries the chapter’s main discipline. A teacher’s score is evidence about ranking; it is not a relevance label. If the teacher ranks a passage first, the passage is preferred by the teacher, not thereby correct. The evaluation must therefore report two different numbers: task relevance against the construction labels, and agreement between teacher and student rankings. The first can be right while the second is low; the second can be high while the first is wrong.

A classifier is the fixed-label special case. Its scorer produces one logit per class, its softmax is a distribution over a known label set, and its cross-entropy is the same negative log-likelihood as the contrastive objective when the candidate set is the class set. The difference is not the algebra; it is the size and openness of the candidate set, the availability of informative negatives, and the evaluation. A classifier’s label set is closed and its decision is a single label; a retriever’s candidate set is the corpus and its quality is an ordering. The contrastive objective transfers because the algebra is shared; the evaluation does not.

Two architectures, two cost profiles

A cross-encoder computes \(s_\theta(q,d)\) from the pair jointly. It can model token-level interactions between query and document, because both are visible to the same forward pass. The price is architectural: for a corpus of \(N\) documents and a query, a cross-encoder needs \(N\) joint forward passes to rank the corpus. A bi-encoder computes \(u_q=f_\theta(q)\) and \(v_d=g_\phi(d)\) separately, then scores with a cheap similarity such as \(u_q^\top v_d\). Document vectors can be precomputed and indexed, so the query-time cost is one query encode plus a similarity search. The bi-encoder gives up joint token interaction in exchange for an index.

This is the design contrast Reimers and Gurevych make concrete for sentence embeddings (P-071): a cross-encoder is accurate but must score every pair, while a siamese bi-encoder amortises document encoding and compares fixed vectors. Their reported BERT-scale timings are theirs, not ours; the architectural point is what this chapter uses. Izacard and Grave apply the same broad idea to question answering, distilling a reader’s attention signal into a retriever so that query–document labels are not required (P-031). Our teacher is a small cross-encoder scorer rather than a reader, and we do not reproduce either paper’s results.

The parameter counts in our study are modest but the asymmetry is real. The cross-encoder teacher has 2,113 parameters: a shared 64-entry token embedding table of width 16, a 32-unit hidden layer over the concatenated query and document vectors, and a scalar output. The bi-encoder student has 576 parameters: the same vocabulary, width 8, mean pooling, an \(8\times 8\) projection, and L2 normalisation. The parameter ratio is 3.67. That ratio alone does not establish an efficiency gain. The architectural difference does: the teacher’s scoring cost grows with the corpus, while the student’s document encoding is paid once and its query encoding once per query.

Different kinds of function, not just sizes

Figure 1. The asymmetry that matters: joint scoring per pair against encode-once plus index search. The 3.67 parameter ratio is arithmetic; the cost structure is the argument. Mechanism schematic.

We measured the student’s query latency, not the teacher’s. At 96 documents, the student’s encode-plus-search step took 0.019–0.027 ms per query across arms and seeds. The teacher would need 96 joint forward passes for the same corpus; we did not time that loop, and we do not invent a number for it. The honest cost claim is therefore conditional: the student has a smaller model, a 3,072-byte document index for 96 eight-dimensional vectors, and a measured tens-of-microseconds query path. Whether that matters at production corpus sizes depends on the index, the hardware, and the candidate-generation stage, none of which this study models.

Embedding geometry

A bi-encoder maps a token sequence to a vector. Mean pooling is the simplest pooling rule: average the token embeddings, ignoring padding, then project. With a projection matrix \(W\) and a pooled vector \(h\), the student produces

\[ u = \frac{W h}{\lVert W h \rVert_2 + \epsilon}, \qquad v = \frac{W h'}{\lVert W h' \rVert_2 + \epsilon}. \]
The L2 normalisation is not decoration. It removes the scale of the vector, so the dot product becomes the cosine similarity:
\[ u^\top v = \cos(u,v) = \frac{u \cdot v}{\lVert u\rVert_2 \lVert v\rVert_2}. \]
Cosine similarity lies in \([-1,1]\) and is invariant to positive rescaling of either vector. Ranking by cosine is therefore ranking by angle, not by magnitude. That is useful when document length or embedding norm varies for reasons unrelated to relevance, and it is also a constraint: the student cannot express "this document is relevant and unusually confident" through a larger norm. If a task needs calibrated relevance scores, a cosine score must be calibrated separately; the ranking metric does not supply that calibration.

The student in this study uses a shared embedding table for queries and documents, mean pooling over non-padding tokens, an \(8\times 8\) linear projection, and L2 normalisation. The teacher uses a separate cross-encoder path: it pools query and document embeddings separately, concatenates them, and passes the pair through a small MLP. The two models do not share a representation space. That matters for what “representation transfer” can mean here. We measure rank transfer through relevance metrics and top-1 agreement; we do not measure representation similarity, because the teacher and student spaces have different dimensions and no alignment map was fitted. A CKA or linear-probe study would answer a different question and was not run.

The contrastive objective

For a query \(q\), let the candidate list be \(C(q)=\{d^+\}\cup N(q)\), where \(d^+\) is a relevant passage and \(N(q)\) is a set of non-relevant candidates. Write the raw similarity \(r_i = u_q^\top v_i\) and the temperature-scaled score \(s_i = r_i/\tau\), where \(\tau>0\) is the contrastive temperature. The InfoNCE loss treats the positive as the correct class in a softmax over the candidate list:

\[ \mathcal{L}_{\mathrm{NCE}}(q) = -\log \frac{\exp(s_+)} {\sum_{j\in C(q)} \exp(s_j)}. \]
The derivation is ordinary cross-entropy. The model produces one logit per candidate; the target is the positive index. The gradient with respect to the scaled score \(s_i\) is
\[ \frac{\partial \mathcal{L}_{\mathrm{NCE}}} {\partial s_i} = p_i - \mathbf{1}[i=+], \qquad p_i = \frac{\exp(s_i)} {\sum_{j\in C(q)} \exp(s_j)}. \]
For the positive, the gradient is \(p_+-1<0\), so gradient descent raises its score. For a negative, the gradient is \(p_i>0\), so gradient descent lowers its score. The magnitude for a negative is its current softmax probability: a negative that already scores well gets a larger push than one that is already far away. That is why hard negatives matter. With respect to the raw similarity, the chain rule adds a factor \(1/\tau\), because \(s_i=r_i/\tau\). The implementation divides raw scores by \(\tau\) before the cross-entropy, so its gradient with respect to \(r_i\) carries that factor.

Take a three-candidate illustration with raw scores \(2,1,0\) and \(\tau=1\). The softmax is \((0.6652,0.2447,0.0900)\), so the loss is \(-\log 0.6652=0.4076\). At \(\tau=0.5\) the scaled scores are \(4,2,0\), the softmax is \((0.8668,0.1173,0.0159)\), and the loss falls to \(0.1429\): a lower temperature sharpens the distribution and makes an already-correct positive cheap. At \(\tau=2\) the scaled scores are \(1,0.5,0\), the softmax is \((0.5065,0.3072,0.1863)\), and the loss rises to \(0.6803\): a higher temperature spreads mass to negatives and keeps the gradient alive longer. The temperature is not a quality knob with a universally best value; it sets the scale at which score differences become decisive.

Negative sampling changes the effective training distribution. The softmax denominator contains only the candidates in \(C(q)\), not the whole corpus. In-batch negatives make the denominator cheap but biased by batch composition. Hard negatives, mined by the teacher, put high-scoring non-relevant documents into the denominator, which increases the loss and the gradient on the examples that matter most. Suppose the positive scores 2.0, two hard negatives score 1.5 and 1.0, and an in-batch negative scores 0.5.

With \(\tau=1\), the positive probability is \(0.4551\) and the loss is \(0.7873\). Remove the two hard negatives and keep only the in-batch negative: the positive probability rises to \(0.8176\) and the loss falls to \(0.2014\). The comparison is illustrative, not a controlled measurement—the denominator changed—but it shows the mechanism: harder candidates make the same positive look less certain, and therefore teach more.

Hard negatives also carry a risk. A “negative” that is actually relevant is a false negative; pushing it down teaches the student to reject a correct answer. Our corpus has exactly one relevant passage per query, so the construction labels do not contain false negatives. The teacher’s mined negatives exclude that passage by construction. The teacher’s own training sampler, however, drew random negatives without excluding the relevant passage; we record that defect in the limitations rather than pretending the teacher was cleaner than it was.

Score distillation from a reranker

A cross-encoder teacher does not have to supply hard labels. It can supply a distribution over the candidate list. For the same candidate list, let \(a_T(q,d)\) and \(a_S(q,d)\) be the teacher and student scores, and define

\[ p_T(d\mid q) = \operatorname{softmax}\!\left(a_T(q,d)/T\right), \qquad p_S(d\mid q) = \operatorname{softmax}\!\left(a_S(q,d)/T\right), \]
and use the temperature-scaled KL objective from Chapter 05:
\[ \mathcal{L}_{\mathrm{KD}}(q) = T^2 D_{\mathrm{KL}}\!\left(p_T(\cdot\mid q)\,\|\,p_S(\cdot\mid q)\right). \]
The gradient with respect to a student score \(a_S\) is \(T(p_S-p_T)\). The teacher is detached; only the student receives gradient. The \(T^2\) factor compensates for the \(1/T\) scale of the logit gradient, as Chapter 05 derives. The teacher's absolute score level is not identified: adding a constant to every teacher score leaves the softmax unchanged. The student can match the teacher's relative ranking on the candidate list without inheriting a calibrated relevance score. That is acceptable for ranking and insufficient for calibration.

A related objective preserves pairwise similarities between documents rather than query–document scores. That is the relational variant Chapter 09 owns; it was not trained here. The teacher signal in this study is query–document score distillation only, so a reader looking for similarity-preserving representation transfer should treat this chapter as a boundary, not a demonstration.

The crucial limitation is the candidate list. The KL is computed over \(C(q)\), not over the corpus. A student can match the teacher’s local distribution on every training list and still rank the full corpus differently, because the lists are a biased sample of the document space. This is not a technicality; it is the mechanism behind the teacher-only failure below. A local score distribution provides no positive anchor and no global comparison across lists.

Ranking metrics

Three metrics appear in the study, and each answers a different question.

Recall@k asks whether a relevant document appears in the top \(k\). With one relevant document per query, it is 1 if the relevant document is in the top \(k\), and 0 otherwise; the reported value is the mean over queries. It is the bluntest metric and the easiest to interpret.

Mean reciprocal rank (MRR) asks how high the first relevant document appears. For one relevant document at rank \(r\), the reciprocal rank is \(1/r\); MRR is the mean over queries. A relevant document at rank 2 contributes 0.5, at rank 5 contributes 0.2, and absence contributes 0.

Normalised discounted cumulative gain at \(k\) (nDCG@k) asks how much relevance-weighted gain appears in the top \(k\), discounted by position. For binary relevance, with \(rel_i\in\{0,1\}\) at rank \(i=1,\ldots,k\),

\[ \mathrm{DCG@}k = \sum_{i=1}^{k} \frac{rel_i}{\log_2(i+1)}, \qquad \mathrm{nDCG@}k = \frac{\mathrm{DCG@}k}{\mathrm{IDCG@}k}, \]
where \(\mathrm{IDCG@}k\) is the DCG of the ideal ordering, with \(\min(|\mathcal{R}_q|,k)\) relevant documents first. With one relevant document, \(\mathrm{IDCG@}k=1\), so nDCG@k is just the discounted gain of the single relevant document. A relevant document at rank 2 gives \(\mathrm{nDCG@5}=1/\log_2 3=0.6309\), with Recall@5 = 1, Recall@1 = 0, and reciprocal rank 0.5. The same ranking therefore looks perfect on Recall@5, mediocre on nDCG@5, and zero on Recall@1. Reporting only one of them would hide the shape of the result.

With one relevant document per query, \(R@1\) and nDCG@1 are numerically identical, because both are 1 exactly when the relevant document is first. The tables below still report both names where the implementation records both; the duplication is a property of binary relevance, not evidence of two independent measurements.

The frozen E2 retrieval study

The corpus is self-created and fixed: 16 entities \(\times\) 6 relations \(=\) 96 passages, 96 queries, one relevant passage per query. The corpus fingerprint is bbc1973817ce79e535d8a90ecbff55110a12a0252d68309b8ab67bc72373c0c8. Each dataset seed splits the 96 query ids into 64 train, 16 validation and 16 test queries. Passages for test queries remain in the index: that is the retrieval task, not leakage. The split is by query id; the test query ids never appear in a training batch. The per-seed test split fingerprints are baf0e9ff…, fdc3842b… and 81131473…; the reconstruction check is recorded in the experiment correction note.

The teacher is a cross-encoder with width 16, hidden size 32, and 2,113 parameters. It trains for 2,000 updates on 64 query–document pairs per update: eight queries, each with one positive and seven random negatives. Its calibration gate is validation Recall@1 at least 0.40; the measured values were 0.8125, 0.5625 and 0.6875, so the gate passed on all three seeds. The teacher’s held-out test ranking is the ceiling we actually had, not an ideal teacher:

Seed Teacher R@1 Teacher R@5 Teacher MRR Teacher nDCG@5
30 0.688 1.000 0.833 0.876
31 0.750 0.938 0.833 0.856
32 0.875 0.938 0.917 0.914

The students are bi-encoders with width 8, an 8-dimensional projection, 576 parameters, and L2-normalised outputs. All three arms start from identical initialisations per seed and run exactly 600 optimiser updates with batch size 16 queries, Adam at 0.01, and weight decay 0.0001. The arms differ only in the training signal:

  • A-contrastive: InfoNCE with \(\tau=0.07\) on the positive, up to eight teacher-mined hard negatives, and in-batch positives.
  • B-teacher: the same contrastive loss plus the teacher score KL with \(T=2\) and weight \(\lambda=0.5\).
  • C-teacher-only: the teacher score KL only, with no relevance labels.

The candidate list for each training query is the positive first, then up to eight hard negatives, then in-batch positives up to a cap of 24. The positive-first convention is what the InfoNCE implementation expects. Each arm selects its checkpoint by its own validation objective after the full 600-update budget; the validation objective therefore differs across arms, a comparison limitation we state rather than hide.

What the measurements show

The per-seed held-out results are:

Seed Arm R@1 R@5 MRR nDCG@5 Agreement@1
30 A-contrastive 0.312 0.625 0.462 0.474 0.250
30 B-teacher 0.375 0.750 0.511 0.556 0.312
30 C-teacher-only 0.250 0.688 0.416 0.461 0.125
31 A-contrastive 0.375 0.750 0.519 0.563 0.312
31 B-teacher 0.375 0.750 0.523 0.563 0.312
31 C-teacher-only 0.062 0.438 0.206 0.234 0.062
32 A-contrastive 0.188 0.750 0.399 0.470 0.188
32 B-teacher 0.188 0.750 0.397 0.468 0.188
32 C-teacher-only 0.250 0.500 0.384 0.375 0.188

The aggregates, with sample standard deviations over three seeds, are:

Arm R@1 MRR nDCG@5 Agreement@1
A-contrastive 0.292 ± 0.095 0.460 ± 0.060 0.502 ± 0.053 0.250 ± 0.062
B-teacher 0.312 ± 0.108 0.477 ± 0.069 0.529 ± 0.053 0.271 ± 0.072
C-teacher-only 0.188 ± 0.108 0.335 ± 0.113 0.357 ± 0.115 0.125 ± 0.062

These are descriptive three-seed values, not significance tests.

Three observations matter more than the means.

First, the teacher is much better than every student. Its nDCG@5 is 0.856–0.914; the best student mean is 0.529. The student did not preserve the teacher’s ranking. It learned a weak retriever from the same data, with some help from the teacher’s scores.

Second, B’s mean advantage over A is not robust. Per seed, the nDCG@5 difference \(B-A\) is \(+0.082\), \(0.000\) and \(-0.003\). The mean gain is \(+0.027\), entirely carried by seed 30. R@1 is \(+0.0625\), \(0.000\), \(0.000\); R@5 is \(+0.125\), \(0.000\), \(0.000\). The predeclared directional hypothesis—B higher than A on mean nDCG@5—is supported on the mean, but the support is one seed and two ties. This is a weak, seed-concentrated association, not a demonstrated advantage, and it would be dishonest to report only the aggregate.

One seed carries the whole gain

Figure 2. Per-seed B−A nDCG@5: +0.082, 0.000, −0.003. The mean (+0.027) is entirely seed 30. Frozen E2 retrieval measurements.

Third, C is below both A and B on every seed’s nDCG@5, and far below on seed 31 (0.234 against 0.563 and 0.563). Teacher scores alone are a poor training signal for this bi-encoder. The result is not that teacher scores are useless—B’s one gain suggests they can help—but that a score-matching objective without a positive anchor does not teach global relevance.

Teacher–student agreement tells the same story from the other side. B’s top-1 agreement is 0.271 on average, A’s 0.250, C’s 0.125. Low agreement does not prove low relevance: A often disagrees with the teacher and still retrieves the right passage some of the time. High agreement would not prove correctness either. The two numbers must be read together, and neither substitutes for the relevance labels.

Why teacher-only fails

The failure of C is the chapter’s cleanest negative result, and the structure of the objective suggests a mechanical explanation, though the experiment does not isolate it. The KL objective compares the teacher’s softmax and the student’s softmax over the same candidate list. If the teacher puts most mass on the positive and the student learns that shape, the student’s score for the positive rises relative to the hard negatives in that list. But the student never sees the positive against the whole corpus; it sees a sampled candidate list. The global ordering across lists is unconstrained. A student can match every local distribution and still fail to put the positive above the other 95 documents.

On this reading, the positive anchor in A and B supplies the missing global signal. InfoNCE’s target is not “be like the teacher”; it is “put this passage above these candidates.” The teacher’s KL in B adds a softer, list-local preference. The combination gives a small gain on one seed. Without the anchor, on the same reading, C learns a shape without a task.

There is a second reason, specific to this corpus. The candidate list contains up to eight hard negatives and several in-batch positives. The teacher’s scores on those lists may be compressed: a cross-encoder can be confident about the positive while its scores on hard negatives are close together. The KL gradient is \(T(p_S-p_T)\); when the teacher’s distribution is nearly uniform over the negatives, the gradient is small. The contrastive loss, by contrast, always has a positive target and a negative set, so it keeps pushing. This is a hypothesis about the observed failure, not a separately measured decomposition; the experiment did not ablate candidate composition. It is stated as a mechanism worth testing, not as a result.

Cost and efficiency

The measured cost ledger is:

Quantity Value
Teacher parameters 2,113
Student parameters 576
Parameter ratio 3.67
Student artefact 2,304 bytes
Document index (96 × 8 float32) 3,072 bytes
Student query latency 0.019–0.027 ms/query
Teacher test latency not measured
Teacher training 3.32–3.47 s per seed
Student training 10.0–15.6 s per arm per seed
Teacher forward calls, B/C 9,984 per seed
Total study wall time 132.1 s
Peak process memory 457.5 MB

Each of B and C makes 9,984 teacher forward calls per seed: 9,600 training calls (600 updates \(\times\) 16 queries) plus 384 validation calls (24 validations \(\times\) 16 queries). A uses no teacher calls. The teacher’s own training uses 128,000 pair scores per seed (2,000 updates \(\times\) 8 queries \(\times\) 8 pairs). These are operation counts, not prices; no API was used and no provider pricing is invented.

The latency number needs its scope stated. It measures the student’s encode of 16 test queries, the encode of 96 documents, and the similarity matrix, divided by 16 queries. It is not an index build, an ANN search, a batch-throughput measurement, or a production corpus. The 3,072-byte index is a toy. The correct conclusion is architectural: the student has a query-time path that does not scale with corpus size in the way the teacher’s joint scoring does. The incorrect conclusion is that this study demonstrates a production speed-up. It does not.

Rights and provenance

The study uses RM-116, VERIFIED_PERMITTED for train_retrieval_distillation_e2, reviewed by the author on 16 November 2026 and due for review on 16 May 2027. The corpus, queries, teacher and student are generated by repository code; no external corpus, model, API, or paid inference is used. Relevance labels are construction facts. The embeddings and cached teacher scores are derived artefacts of that corpus, and the rights record forbids redistribution beyond the study artefacts. No personal data is present.

A retrieval corpus is not automatically free to use just because it is downloadable. A compiled database can attract a sui generis right under Directive 96/9/EC for substantial investment in obtaining, verifying, or presenting its contents (L-026). This chapter’s corpus is synthesised, so no third-party database is extracted; the record still exists because the next corpus may not be synthetic. The legal finding is dated and action-specific; it is not legal advice, and it does not resolve the rights of a future corpus.

Limitations

The study is one synthetic corpus, three seeds, tiny models, and a single relevant passage per query. Binary relevance makes nDCG@1 and R@1 identical and removes graded-relevance questions. The contrastive temperature (0.07), KL weight (0.5), KL temperature (2.0) and hard-negative depth (8) are single fixed values; no sweep was run. The teacher’s random-negative sampler did not exclude the relevant passage, so some teacher training pairs were contradictory; the teacher’s measured test ranking is the quality actually obtained, and the effect of that label noise on B and C is not measured.

Validation candidate lists omit in-batch negatives, unlike training lists, so each arm’s checkpoint selection distribution differs from its training distribution. The teacher’s latency was not measured. No out-of- distribution query set was evaluated; the held-out split is a query split, not a distribution shift. No representation-similarity or linear-probe analysis was run, so “representation transfer” is not measured. No external replication and no independent review exist.

Three record-level defects are also disclosed rather than buried. RESULTS.md prints the training-query count (64) under an N-q column while the metrics use 16 held-out test queries; the chapter reports 16. The runner mined hard negatives from train and validation queries, not train only as the protocol sentence says; test queries never entered training, and the separately computed test hard-negative list was unused. The protocol names a superseded run that is not present in the repository; the cited run’s manifest and source hashes are intact. These are reporting and preservation defects, not changes to the measured values. They are recorded in experiments/E2-retrieval/results-correction-1.md and planning/00-dfp-11-run-history-gap.md.

Chapter 08 owns feature alignment and Chapter 17 owns the instruction-model baseline whose fixed-update discipline this study follows; this chapter applies those ideas to a retrieval scorer without re-teaching them. Chapter 29 extends the question to multimodal embeddings, Chapter 34 owns ranking evaluation, and Chapter 37 owns corpus copyright.

What comes next

The chapter opened with the difference between choosing a label and ranking a corpus. It closes with the difference between matching a teacher and retaining a task. The student’s relevance improved only weakly, and only on one seed; the teacher’s ranking was not transferred; the teacher-only arm failed. The next chapter changes the output object again. Code generation must produce a structured artefact that compiles or executes, so correctness is not a ranking over candidates but a property of the generated program. Chapter 25 asks what supervision, validation, and evaluation look like when the student’s output must run.

References

Izacard, G. and Grave, E. (2021). Distilling Knowledge from Reader to Retriever for Question Answering. International Conference on Learning Representations (ICLR). arXiv:2012.04584. https://arxiv.org/abs/2012.04584

Reimers, N. and Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of EMNLP 2019. arXiv:1908.10084. https://arxiv.org/abs/1908.10084

Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases. In force 27 March 1996. https://eur-lex.europa.eu/eli/dir/96/9/oj (retrieved 12 November 2026). Cited as L-026 for the sui generis database-right context; not legal advice.