Continual and Adaptive Distillation
Can a student keep learning from new teachers without forgetting what it already knew? A bounded two-task E1 study comparing sequential fine-tuning, bounded replay, and distillation regularisation — with retention policy, deletion, rollback, and a failed joint-training reference.
Chapters 17 through 22 trained a student once, on one distribution, and evaluated it once. Real deployments do not work that way: teachers are refreshed, data arrives in batches, tasks change, and someone eventually asks for an example to be deleted. This chapter asks the operational question:
How can a student keep learning from new teachers or data without forgetting what it already knew — and what must be retained or deleted?
A student is a function of its training history, and continual learning changes the history while the function is still in use. Every other chapter in this book could pretend training happens once, in a closed room, with the test set waiting outside. This one cannot: the room stays open, new teachers walk in, old data is supposed to leave, and the evaluation must cover both what was learned and what was kept. The honest answer, on a two-task spiral study with tiny MLPs, is a ranking with a surprise at both ends. Sequential fine-tuning forgets catastrophically (Task-A accuracy on seed 40 falls from 0.73 to 0.19; the mean after Task B is 0.18).
Bounded replay of 200 old examples retains the most (0.48) at the largest cost to the new task (0.42 vs 0.76 sequential). Distillation regularisation against the frozen old teacher sits between the two (0.43 retained, 0.48 new). And the joint-training “upper bound” fails both tasks (0.44/0.41), most plausibly because the tasks contradict each other by design (Task B reassigns Task A’s labels), and no training schedule makes contradictory supervision consistent. Forgetting is real, measurable, and mitigable; the reference that was supposed to bound the mitigation is itself a casualty. That is the chapter’s finding, and the machinery below exists to make it checkable.
What counts as continual
A continual setup has three ingredients, and missing any one of them reduces it to ordinary training. First, a sequence: the student trains on Task A, then on Task B, with no return to A’s training data except through an explicit mechanism. Second, a frozen past: the stage-A checkpoint G0 is pinned by checkpoint hash and never updated again; anything that claims to “preserve” A must name the pinned version it preserves. Third, a retention policy: which old examples may be kept, for what purpose, and until when — plus a deletion path that actually purges them. Our replay buffer carries exactly that metadata (purpose: “retain task A”, retained-until: “study-end”), and a test proves deletion works by deleting.
Without the buffer, “replay” is just retraining; without the pin, “the old teacher” is whichever checkpoint someone happened to keep; without the policy, retention is hoarding. Walk through an erasure to see why all three matter together: a request arrives to delete Task-A example #417. The buffer finds its live records, marks them retired, and the post-deletion audit reads 199 live, 1 retired — and every subsequent batch is drawn from live records only, so the example can never silently re-enter training through a stale index. Without versioning, the same request would leave the teacher untouched — G0 was trained on #417 and its weights still encode it — which is exactly why the chapter distinguishes deletion of data (implemented, tested) from deletion of influence (not claimed, and flagged as the harder problem it is).
KD-016, KD-026, KD-041 — no new concept
Chapter 23 introduces no new concept, deliberately. The distillation loss is KD-016, the same mixed objective as Chapters 05 and 21. The frozen teacher is KD-026 self-distillation machinery: G0 supervises its own successor. Provenance and retention are KD-041: every candidate, checkpoint, and buffer row carries identity, source, and rights status. What is new is the schedule — old supervision applied during new training — and the accounting: forgetting, transfer, replay volume, teacher accesses, rollback. A chapter that needed a new concept for “train, then train again” would be inventing vocabulary to hide a thin experiment. The vocabulary we have is sufficient; the measurements are what is missing, so the chapter measures.
Eight ways to supervise, compared
Before the measurements, the map. Every method in Part IV answers the same question — where does the student’s target come from? — differently:
| Approach | Teacher identity | Label source | Data source | Soft vs hard | New unlabelled inputs? | Teacher frozen? | Student generates states? | Main failure mode |
|---|---|---|---|---|---|---|---|---|
| Supervised (A) | none | ground truth L | L | hard | no | n/a | no | label noise ceiling |
| External KD (Ch06/21-E) | big MLP, other arch | ground truth + big-teacher soft | L | both | no | yes | no | confident teacher adds nothing |
| Born-again (Ch21-B) | frozen G0, same arch | ground truth + G0 soft | L | both | no | yes | no | inherits teacher bias softly |
| Pseudo-label self-training (Ch21-C) | frozen G0, same arch | ground truth + G0 hard on U | L + U | hard | yes | yes | no | confident errors amplified |
| Hard-teacher control (Ch21-D) | frozen G0, same arch | G0 hard on L only | L | hard | no | yes | no | echo chamber, no grounding |
| On-policy (Ch20) | frozen external teacher | teacher conditionals on student prefixes | evolving student states | soft | n/a (states, not labels) | yes | yes | staleness, sampling cost |
| Continual replay (this chapter, B) | frozen G0, same arch | ground truth A (replayed) + ground truth B | old + new | hard | no (old labels reused) | yes | no | interference, retention debt |
| Continual distillation (this chapter, C) | frozen G0, same arch | ground truth B + G0 soft on B | B only | both | no | yes | no | obsolete-teacher drag |
Read down the “New unlabelled inputs?” column: only self-training adds inputs the labels never described. Read across “Soft vs hard”: only born-again, external KD, and continual-distillation soften. The table is the chapter’s terminological contract — every later claim cites a row.
Forgetting, measured
Forgetting is not a feeling about the loss curve. It is a number: accuracy on the retained task before stage 2 minus accuracy after, \(F = A_{before} - A_{after}\), with the sign carrying the news — negative means backward gain.
On our permuted-label pair, plain sequential fine-tuning (Arm A) forgets 0.537/0.330/0.509 per seed (G0 at 0.730/0.540/0.655, then 0.193/0.210/0.146). The mean, 0.459, is not a subtle effect wearing statistical camouflage; it is a collapse to near-chance on a four-class task. Report it per seed, with the denominator stated (1000 held-out Task-A examples), and resist the urge to average it away: seed 41’s G0 was weaker to begin with (0.540), so its forgetting is smaller in absolute terms while identical in kind. Forgetting is always relative to what was known.
Forward and backward transfer
Transfer is forgetting’s mirror, measured the same way. Forward transfer asks whether the old capability helped the new one: Arm A (B-starting-from-G0) minus Arm E (fresh init), both evaluated on Task B. Per seed: +0.068, +0.052, +0.110 — small, positive, consistent in sign. The old weights are a better starting point than random ones even though the labels were permuted; features transfer where mappings do not. Read that twice, because it bounds the forgetting story: the same G0 initialisation that loses Task A (0.46 mean forgetting) accelerates Task B (+0.08 mean transfer).
Forgetting the mapping while keeping the geometry is not a contradiction — it is what “representation reuse with label reassignment” looks like from the inside, and it is why E-scratch (0.678 B mean) trails A-seq (0.755) on every single seed despite never having seen Task A at all. Backward transfer would be new training improving the old task; it does not occur here (all forgetting values are positive), and the chapter does not invent an instance. Negative forward transfer — the new task made harder by the old start — does not occur either, which is itself a finding worth one sentence: permuted labels scramble the mapping but reuse the geometry, so the student starts closer and still forgets what the geometry was for.
Replay: the bounded buffer
Arm B keeps 200 Task-A examples (seeded subset, 1/6 of A’s data), each tagged with task, seed, purpose, and retention window, and mixes 32 replay rows into every 64-example batch — 96,000 replayed presentations over 3000 updates, derived from the recorded batch composition (32 replay rows × 3,000 updates) rather than read from a counter. The buffer is bounded (capacity 200; the audit reports live/retired/evicted), deletion is implemented and tested (a two-record deletion purges both indices from all subsequent batches; the post-deletion audit reads 198 live, 2 retired), and overlap checks forbid test examples from ever entering it.
This is the entire difference between replay and quiet retraining: the buffer is small, labelled, deletable, and audited. What it buys: the best retention in the study (0.481 mean A accuracy). What it costs: the worst new-task accuracy among honest arms (0.424 vs 0.755 sequential) — replayed contradictions poison every batch, and the chapter says so plainly.
Distillation regularisation
Arm C adds the frozen G0’s soft targets on the new task’s inputs to the ordinary new-task loss — the Learning-without-Forgetting pattern (P-056): \(\lambda T^2 D_{KL}(p_{G0} \mid\mid p_S)\) with \(\lambda = 0.5\), \(T = 2\), computed on Task-B rows, gradients into the student only. The teacher never sees Task A again; it scores Task-B prefixes, and the student is penalised for drifting from what G0 would have said there.
Result: 0.430 retained, 0.478 new — between replay and sequential on both axes, as the mechanism predicts (softer constraint than replay’s hard labels, stronger than nothing). The implementation is twelve lines on top of the shared fixed-update loop, and the chapter keeps it that way: regularisation that needs a framework to explain is usually regularisation that does not work.
EWC, distinguished without equivalence
Elastic weight consolidation (Kirkpatrick et al., P-070) is the obvious comparison: penalise changes to weights that mattered for the old task, measured by Fisher information, instead of distilling the old outputs. The chapter does not implement it, and says why in one paragraph: EWC and distillation regularisation answer different questions. EWC asks where in parameter space the old solution lives and charges rent for leaving; LwF-style distillation asks what the old model did and charges for behavioural drift. They coincide when behaviour determines parameters uniquely and diverge everywhere else — which, in overparameterised networks, is everywhere. No equivalence is claimed, no Fisher matrix is computed, and the reader is warned that “regularisation” is a family name, not a method.
Example A — two tasks, one student, frozen history
Concretely: Task A is a 4-class spiral (1200/400/1000 train/val/test, noise 0.08, seeds 40/41/42). Task B is the same generator at seed+5000 with class labels permuted (0→1→2→3→0) — same inputs distribution, contradictory mapping, the permuted-task analogue of the classic continual benchmark. G0 is a 2→32→4 MLP (228 parameters) trained 3000 updates on A, frozen, checkpoint-hashed (G0 test accuracy 0.730/0.540/0.655 — measured, not tuned; seed 41’s draw is genuinely harder and the chapter neither hides nor smooths it). Stage 2 runs 3000 matched updates per arm with seed-matched inits: A/B/C continue from identical frozen G0 weights (asserted); D/E start from identical fresh inits (asserted). Every number that follows is a held-out measurement against that frozen history.
Example B — the LwF-style objective, numerically
Take a single Task-B input where frozen G0 emits logits [2.0, 0.5, −1.0, −0.5] and the student (mid-training, still close to G0) emits [1.8, 0.6, −0.9, −0.6]. With \(T = 2\): tempered teacher ≈ [0.426, 0.260, 0.149, 0.165], tempered student ≈ [0.410, 0.266, 0.153, 0.170] — the KL term is small (~0.001), correctly reporting “no drift yet”.
Now move the student far: [−1, 3, 0, 0] — tempered ≈ [0.09, 0.67, 0.12, 0.12] — and the term grows past 0.5 × \(T^2\) × 0.5 ≈ 1.0, dominating the hard loss. The arithmetic is the mechanism: small drift, small penalty; large drift, large penalty; temperature setting the scale at which “large” begins. The test suite pins this behaviour on hand-computed values (test_lwf_*) so the runner cannot silently redefine it.
Figure 1. Example B’s two cases: KL ~0.001 when the student sits on the teacher, ~1.0 when it moves far. Worked arithmetic, test-pinned.
Example C — the forgetting table
The chapter’s central table, per seed (A-test / B-test):
| Seed | Arm | Acc-A | Acc-B |
|---|---|---|---|
| 40 | G0 (before) | 0.730 | — |
| 40 | A-seq | 0.193 | 0.755 |
| 40 | B-replay | 0.483 | 0.409 |
| 40 | C-distill | 0.398 | 0.482 |
| 40 | D-joint | 0.447 | 0.356 |
| 40 | E-scratch | 0.216 | 0.687 |
| 41 | G0 (before) | 0.540 | — |
| 41 | A-seq | 0.210 | 0.722 |
| 41 | B-replay | 0.442 | 0.422 |
| 41 | C-distill | 0.372 | 0.457 |
| 41 | D-joint | 0.419 | 0.473 |
| 41 | E-scratch | 0.192 | 0.670 |
| 42 | G0 (before) | 0.655 | — |
| 42 | A-seq | 0.146 | 0.788 |
| 42 | B-replay | 0.519 | 0.440 |
| 42 | C-distill | 0.519 | 0.494 |
| 42 | D-joint | 0.456 | 0.410 |
| 42 | E-scratch | 0.192 | 0.678 |
Aggregates: A-acc A-seq 0.183 ± 0.033, B-replay 0.481 ± 0.039, C-distill 0.430 ± 0.078, D-joint 0.441 ± 0.019, E-scratch 0.200 ± 0.014; B-acc A-seq 0.755 ± 0.033, B-replay 0.424 ± 0.016, C-distill 0.478 ± 0.019, D-joint 0.413 ± 0.059, E-scratch 0.678 ± 0.009. Read it twice: once for what each arm kept, once for what each arm paid.
Per-seed forgetting for sequential training is 0.537/0.330/0.509 (G0 at 0.730/0.540/0.655, A-seq ending at 0.193/0.210/0.146); forward transfer over fresh starts is +0.068/+0.052/+0.110. G0 itself is no champion — seed 41’s teacher tests at 0.540 — and the chapter does not need it to be: every comparison that matters is within-seed, against that seed’s own frozen G0, and the replay arm never even looks at G0’s outputs (ground-truth replay labels only).
Actual optimiser and token budgets
All 15 stage-2 students ran exactly 3000 updates, batch 64; G0 and the fresh-init baselines are asserted bit-identical across arms within each seed. Data exposure differs by design and is reported, not matched: replay sees 96,000 extra A presentations; joint sees 2400-example pools per update; distillation sees no extra inputs at all — only G0’s soft targets on B rows, through one 64-row G0 forward per update (3,000 training forwards per seed) plus a single 400-row validation cache. Wall time is minutes total (79.5 s frozen run); the chapter lists per-arm seconds from the manifest rather than pretending CPU seconds transfer across machines. Matched updates plus disclosed exposure: that is the Ch15 discipline, applied, not cited.
Replay versus regularisation, head to head
B versus C is the chapter’s designed comparison: same starting weights, same new-task batches, same budgets — differing only in whether old behaviour is enforced with hard replayed labels or soft frozen-teacher targets. Replay retains more (0.481 vs 0.430, all three seeds) and learns the new task worse (0.424 vs 0.478, all three seeds).
The mechanism is visible in the batches: replay injects ground-truth old contradictions into every step, which anchors A strongly and fights B learning on every step; the KL term only resists drift, never asserting old labels. Neither dominates; they trade retention against plasticity at different rates, and the chapter refuses to average the trade into a single “winner”. If you must pick one number, pick forgetting reduced: B saves 0.30 of A’s accuracy over sequential, C saves 0.25 — with C costing no storage and no retention obligations.
Figure 2. Arm means on the two axes that matter: replay retains most and learns least; nobody leads both. Frozen E1 continual measurements.
The joint-training failure
Arm D was supposed to be the reference: train on everything, bound what retention can recover. It failed both tasks (0.441/0.413) — worse than replay on A, worse than scratch on B, on all three seeds. The most plausible reason is not mysterious once stated: A and B contradict each other on statistically identical inputs, so the joint objective has no solution that serves both, and 3000 updates split the difference into mutual mediocrity. Note the variance signature: D’s B-accuracy spread (±0.059) is the widest in the study — compromise solutions are unstable across seeds, while the single-task arms (A-seq B ±0.033, E ±0.009) are comparatively tight.
This is evidence against the “joint training is the upper bound” hypothesis for contradictory task pairs, and the chapter states the scope exactly that way: no claim about complementary tasks, where joint training routinely works. A failed reference is still evidence — it tells you the regime you are in. Here the regime is interference, and the honest upper bound is not joint training but the per-task specialists: 0.73-ish on A (G0), 0.69-ish on B (E). Nothing in the study beats specialists at their own task; the question was only what survives the transition.
Teacher drift and version pinning
A frozen teacher is a versioned teacher, or the word means nothing. The runner records G0’s checkpoint hash and asserts it before every stage-2 use; a refreshed teacher would be a new version, never a silent replacement. To show why the distinction matters, a diagnostic retrains G0’s recipe from a different seed (G0′) and measures agreement on Task-A test inputs: 0.803/0.577/0.676 per seed. Same recipe, same data distribution, different init — and up to 42% of predictions disagree.
That spread is itself a finding: the “teacher” in a continual study is not a function but a sample, and retention measured against an unversioned moving target is uninterpretable. Had the teacher been refreshed mid-study without versioning, every “retention” number would be relative to which teacher — the audit costs one extra training run and settles the question that sinks unversioned continual studies before they start. Note what the drift numbers do not show: they do not show G0′ is better or worse (its accuracy was not the question), only that “the same procedure” is not “the same teacher”.
Rollback, verified not assumed
Rollback is a procedure, not a promise: reload the pinned G0 checkpoint into a fresh model object, re-evaluate on the Task-A test split, and assert equality with the recorded G0 accuracy. The runner does exactly this, with zero tolerance beyond floating-point identity — and it passes on all three seeds (0.730/0.540/0.655 reproduced exactly).
This is the cheapest strong claim in the chapter: nothing about learning, everything about engineering. A system that cannot restore its previous self cannot claim to have “kept” anything; ours can, and the test that proves it runs in the standard suite, not just in the experiment script. Note the asymmetry the rollback exposes: restoring weights is trivial and exact, while restoring behaviour after stage-2 training is the entire unsolved problem — the checkpoint is a time machine for parameters, not for knowledge.
Cost accounting
G0 training ~3.2–3.5 s per seed; stage-2 arms ~3–5 s each; teacher G0′ diagnostic one more G0-scale run; total frozen wall time 79.5 s, peak process memory recorded per run (host-specific, not a capability claim). Replay storage: 200 rows × (2 floats + 1 int) — negligible here, and the chapter says so instead of inflating it into a trade-off: at this scale the real replay cost is interference (B-task accuracy 0.424 vs 0.755), not bytes. Teacher queries for Arm C: 3,001 forward calls per seed — one 64-row G0 forward per update (3,000) plus a single 400-row validation cache.
The manifest records wall time but not a teacher-forward counter, so this count is derived from the frozen loop, not read from a field. No provider pricing exists; none is invented. The marginal question the ledger answers: retention cost B ≈ 96k replay presentations for +0.30 A points and −0.33 B points; distillation cost ≈ 3,001 teacher forward calls for +0.25/−0.28. Whether either trade is worth it depends on which task pays the bills — the chapter supplies both sides.
Failure modes, honestly listed
What would break each arm, stated before anyone asks. A-seq: any task change at all — it has no retention mechanism, so forgetting is not a risk but a schedule. B-replay: buffer poisoning (wrong labels in, wrong behaviour out), retention violations (an undeleted example is a promise broken — the deletion test exists because the failure mode is real), staleness (a buffer frozen while the world moves on).
C-distill: teacher incompetence (a bad G0 regularises toward bad behaviour — our G0s are mediocre, and the 0.43 retention ceiling partly reflects that); threshold-free distillation with no selection. D-joint: contradiction, as measured. E-scratch: nothing can break it except the task itself — it is the floor, and floors are load-bearing. A failed seed would be reported, not dropped; all three seeds completed, so there is nothing to disclose beyond the spread, which the tables already show.
What else moved, for the record, since accuracy alone invites selective reading. NLL on Task A (seed 40): A-seq 2.76, B-replay 1.14, C-distill 1.14, D-joint 1.23, E-scratch 2.81 — the retained arms are also the better-calibrated likelihoods on the old task, while the sequential and fresh arms are confidently wrong there.
ECE on Task A tells the same story in reverse (A-seq 0.41–0.49, B/C/D 0.14–0.22, E 0.33–0.37): forgetting shows up as miscalibration before it shows up as anything else, because the model keeps its new-task confidence on old-task inputs. Agreement with frozen G0 on Task A collapses everywhere — 0.08–0.21 for A-seq itself, which started as G0 — proving that parameter inheritance is not behavioural inheritance: three thousand updates later, nothing remains of the starting point except the initialisation nobody can see.
Rights, provenance, and limits
Every replay row carries task, seed, purpose, and retention window;
every teacher carries a checkpoint hash and a freeze attestation;
every student carries its init source (G0 weights vs fresh seed)
and exact update count; RM-115
(train_continual_distillation_e1, verified-permitted, gate first)
covers the whole study including retention and deletion — the first
rights record in this book whose action explicitly includes erasure
semantics, because continual systems are the first systems in this
book that must forget on purpose. Nothing is redistributed.
Limitations, without softening: one synthetic task pair, three seeds, tiny MLPs; permuted labels are strong interference by design (pilot-verified), so milder drift regimes are unmeasured; single replay size (200), single λ (0.5), single T (2); EWC-style weight penalties not implemented (conceptual contrast only, no equivalence claimed); no external replication, no independent review. The strongest caveat is also the most interesting: all retention mechanisms were tested against a contradictory task, where any retention fights learning — the rankings may differ where tasks merely differ.
Preparing for retrieval
Continual learning asked what survives when the teaching changes; retrieval asks what survives when the representation must serve a new purpose. Chapter 24 takes a ranking teacher — a cross-encoder that scores query and document together — and distils it into a bi-encoder student that must embed them separately, preserving ranking behaviour without the teacher in the loop.
It inherits the version pinning (the teacher is frozen and hashed), the cost ledger (plus index size and query latency), the rights discipline (a corpus has owners in a way spirals do not), and the warning this chapter earned twice over: matching yesterday’s behaviour is not the same as keeping yesterday’s capability, and only held-out measurement — on the retained task, not just the new one — tells the difference.
References
Li, Z. and Hoiem, D. (2017). Learning without Forgetting. ECCV 2016 (TPAMI version). https://arxiv.org/abs/1606.09282
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D. and Hadsell, R. (2017). Overcoming Catastrophic Forgetting in Neural Networks. PNAS 114. https://arxiv.org/abs/1612.00796