← Distillation From First Principles

Vision, Speech, and Multimodal Distillation

What transfers between modalities, and when does preserving an embedding relationship fail to preserve capability? A bounded image-to-signal E2 extension where relational supervision shapes representations without moving accuracy.

Chapter 28 routed between experts that shared an observation space. This chapter removes that comfort: the teacher sees images, the student hears only signals — sixteen numbers that encode the same four classes through a different sense. Nothing lines up by construction. The dimensions differ, the invariances differ, and no coordinate in one space means anything in the other:

What information can be transferred between modalities, and when does preserving an embedding relationship fail to preserve the underlying capability?

On our bounded study, the answer is precise and narrow. A signal student supervised with the image teacher’s relational structure matches or exceeds the supervised baseline on cross-modal matching and holds the lowest relational distance on all three seeds — while accuracy ties near ceiling everywhere (0.991 vs 0.991 vs 0.988) and a shuffled-correspondence control matches on accuracy while failing on relations. Relations transfer without moving capability, and capability does not certify relations. The cross-modal analogue of the book’s running distinction, measured on both sides.

Two senses, one label

The task is four-way classification. Modality A is a 12×12 image of a shape: horizontal bar, vertical bar, down-right diagonal, or down-left diagonal, positioned randomly with Gaussian noise. Modality B is a 16-dimensional signal: the class prototype (a scaled one-hot in the first four dims) plus Gaussian noise.

The pairing is by shared label within a split — 600 train, 200 validation, 400 held-out test pairs per seed — and the pairing is the entire bridge between the senses. There is no text, no speech, no photograph of anything: the signal is an audio-like stand-in (a vector per example, no temporal structure), and the chapter states that limit plainly. What transfers here is the correspondence pattern, not any fact about real spectrograms.

The teacher is an image convolutional classifier (16→32 channels, ~5k parameters) trained on the images; it scores 1.000 on validation and test on all seeds — the positive control the protocol gates on, passed everywhere. The students are signal MLPs (16→32→16→4, 1,140 parameters) that never see an image at any stage: not in training, not in validation, not at test. That absence is the missing-modality condition, and it is structural rather than incidental. A deployment that distils an image teacher into a signal-only student — a sensor that cannot carry a camera — evaluates exactly this shape: train with both senses, deploy with one.

That asymmetry dictates the encoder designs, and the designs are not interchangeable. The CNN exploits spatial locality: 3×3 kernels, max-pooling, translation-equivariant features — the right bias for bars and diagonals at random positions. The signal MLP has no such bias: every input dim meets every first-layer weight, and order among the 16 signal dims carries no meaning the architecture respects. The same cross-entropy loss therefore does different work in each: for the CNN it tunes spatial filters, for the MLP it tunes a lookup table with smoothing.

The information bottleneck differs accordingly — 32 pooled image channels against a 16-dim signal embedding — and the RKD term is the only supervision that compares them without pretending they are the same kind of object. On time: neither modality here has temporal structure (one image, one vector per example), so the temporal-vs-spatial question the contract raises is unmeasured by design. A real speech encoder — frame-level features, masking, layer-wise targets of the DistilHuBERT kind (P-052) — would add exactly the sequential dimension this study omits, and the chapter claims nothing about it.

Preprocessing is part of the method and is recorded as such. Images are clamped to [0, 1] with noise 0.10; signals are standardised with train-split mean and standard deviation (per-dim, floor 1e-6), and every later split — validation, test, and the doubled-noise robustness variant — is transformed with the train statistics, never its own.

Fitting normalisation on the test set would leak its distribution into the features; the runner fits once, on train, and the manifest records the recipe rather than the numbers. Swap the recipes — standardise the images, clamp the signals — and nothing breaks loudly, but the image mean absorbs the near-zero background while signal outliers survive unscaled; the manifest exists so that a replication which “just normalises everything the same way” can be diagnosed when its numbers differ.

Why raw feature matching is ill-posed

The teacher’s pooled representation lives in \(\mathbb{R}^{32}\); the student’s embedding lives in \(\mathbb{R}^{16}\). The equation \(h_S \approx h_T\) is not even type-correct, and padding or truncating coordinates to force it would assert a correspondence between the teacher’s 17th filter and the student’s first hidden unit that nothing justifies. This is the modality gap in miniature: different sensors, different encoders, different invariances, no shared axes.

The standard repair is a learned projection. With a projection \(W\), the feature-alignment objective is

\[ \mathcal{L}_{\mathrm{align}}(x) = \lVert W h_S(x) - h_T(x) \rVert_2^2, \qquad\text{or}\qquad 1 - \frac{\langle W h_S(x), h_T(x)\rangle} {\lVert W h_S(x)\rVert_2\,\lVert h_T(x)\rVert_2}, \]
mean-squared or cosine form (Chapter 08 derives both and the projection mechanics). The projection solves the type error, but it does not solve the semantic one: \(W\) can minimise the loss by aligning coordinates while the spaces disagree about everything downstream — the "treating a projection as semantic alignment without evidence" failure the contract warns against. A small projection error proves the student can be linearly mapped onto the teacher's coordinates, not that it sees what the teacher sees. Our study does not train a projection for exactly this reason; the equation is derived so the reader understands the rejected alternative, and the experiment uses relations instead.

A worked pair makes the three objectives’ different demands concrete. Take a student vector \((1, 0)\) and a teacher vector \((0.9, 0.1)\) with \(W = I\): the MSE is \((0.1)^2 + (0.1)^2 = 0.02\) and the cosine similarity is \(0.9/\sqrt{0.82} \approx 0.994\) — near-perfect by either alignment measure, yet the two vectors could still induce different pairwise relations to a third point, which is exactly what the RKD term below checks and the projection losses do not.

Near-perfect alignment that sees nothing

Figure 1. MSE 0.02 and cosine distance 0.006 for the worked pair: both declare alignment while relations go unchecked. Worked arithmetic.

Contrastive alignment, derived and bounded

The contrastive alternative avoids coordinates by comparing similarities. Given paired embeddings \((u_i, v_i)\) — student and teacher views of the same example — with cosine similarity \(s_{ij} = u_i^\top v_j / \tau\) over L2-normalised vectors, the paired InfoNCE loss is

\[ \mathcal{L}_{\mathrm{NCE}} = -\frac{1}{N}\sum_{i=1}^{N} \log\frac{\exp(s_{ii})}{\sum_{j=1}^{N}\exp(s_{ij)}}, \]
with an optional symmetric term swapping the two directions (the CLIP design, Radford et al., P-051, uses symmetric contrastive supervision over 400 million image–text pairs to learn transferable visual models — background for the framing here, not a method we reproduce or a result we borrow; the scale differs from ours by six orders of magnitude (400 million vs 600), the supervision is open vocabulary where ours is four closed classes, and the chapter draws no comparison between the two beyond the shared contrastive form.

Similarly, DistilHuBERT (Chang et al., P-052) distils a HuBERT speech model layer-wise into a model 75% smaller, retaining most performance across ten tasks — the speech-distillation precedent, cited for the layer-wise supervision design, not reproduced and not evidence about our prototype signals, which lack frames, masking, and temporal structure entirely. Both papers anchor the chapter’s scope sentence: real image–text and speech representation distillation exist, are peer-reviewed, and operate at scales and with structures this study does not attempt. The normalisation is load-bearing: without it, the objective can be minimised by scaling magnitudes rather than aligning directions, and the implementation normalises inside the loss so the chapter’s equation and code cannot disagree.

A worked three-candidate check, paralleling Chapter 24’s: raw scores \(2, 1, 0\) at \(\tau = 1\) give softmax \((0.665, 0.245, 0.090)\) and loss \(-\log 0.665 = 0.408\); the gradient pushes the paired score up and the unpaired scores down in proportion to their current probabilities. The batch composition caveat from Chapter 24 applies unchanged: the denominator contains only in-batch candidates, so “contrastive” here means discriminating among the batch, not among the world — one more reason the helper is tested but not trained as an arm, since a contrastive arm would measure batch construction as much as modality transfer.

Our workbench implements this loss, tests it (matched pairs score below shuffled pairs; temperature and shape refusals pinned), and does not train on it. The trained objective is relational, for a reason the next section gives: with 400 test pairs and a saturated accuracy metric, the question is whether the teacher’s similarity structure transfers, and relations state that question directly.

Relations across the gap

Relational distillation (Chapter 09) compares examples to each other rather than to coordinates. For a batch with student embeddings \(S\) and teacher embeddings \(T\), the RKD distance term used here is

\[ \mathcal{L}_{\mathrm{RKD\text{-}D}} = \mathrm{Huber}_\delta\!\left( \frac{D(S)}{\bar D(S)}, \frac{D(T)}{\bar D(T)} \right), \]
meaned over off-diagonal pairs, where \(D(\cdot)\) is the pairwise Euclidean distance matrix and each is normalised by its own off-diagonal mean. Chapter 09 derives the invariances: each side's normalisation makes the term invariant to a common positive scale in either space, and Huber (\(\delta = 1\)) caps outlier pairs. The property that matters across modalities is what the loss does not contain: no projection, no shared dimension, no coordinate map. A 32-dim image space and a 16-dim signal space have comparable pairwise structures even though no vector in one equals any vector in the other.

Relations are the modality-agnostic carrier, and this study is the test of whether the carrier delivers. The paired-data assumption behind all three objectives deserves one explicit paragraph, because it is the assumption most likely to fail silently elsewhere. Every loss here — projection, contrastive, relational — assumes each training item exists in both senses with the same underlying label: the image and the signal are two views of one fact. Break that, and each objective fails differently: the projection aligns to noise, the contrastive loss treats true pairs as negatives, RKD transfers another batch’s structure.

Arm C is the controlled break (permuted pairing, same marginals), and its RKD rising above B’s on every seed while accuracy ties is the signature of a broken correspondence that accuracy cannot see. What the study never tests is the harder break: senses that disagree about the label itself — an image of a bar paired with the signal for a diagonal. Both modalities here encode the same four classes by construction, so a true modality gap, where no correspondence exists to transfer, never occurs. The chapter’s claims stop at the pairing it has.

Example A — the paired batch

Take four training items, one per class. The images are four 12×12 noisy shapes; the signals are four 16-vectors near their class prototypes. The teacher embeds the images (32-dim GAP vectors); the student embeds the signals (16-dim). The \(4\times4\) teacher distance matrix records which shapes the teacher finds similar — diagonals closer to each other than to bars, say — and the student’s RKD term pulls its own \(4\times4\) matrix toward the teacher’s, each normalised by its own mean.

Arm B trains on true pairs (image \(i\) with signal \(i\)); arm C trains on a seeded permutation (image \(\pi(i)\) with signal \(i\)), destroying the correspondence while keeping every marginal distribution identical. If pairing is irrelevant, B and C should tie on relations; if the teacher’s structure matters, B should sit below A (no relations) and C above it. The frozen result confirms half of that: on all three seeds B is lowest and C sits above B, but C and A are indistinguishable (C is below A on two seeds), with accuracy tied throughout.

Example B — the missing modality, twice over

The student never sees images: that is missing-modality condition one, structural and total. Missing-modality condition two is evaluative: the doubled-noise test regenerates the test signals with noise 1.0 instead of 0.5 (same labels, same train-fitted standardisation) and re-evaluates. Accuracy falls everywhere — A 0.991 → 0.787, B 0.991 → 0.799, C 0.988 → 0.786 — and B’s small edge over the shuffled control persists (+0.012/+0.010/+0.018 per seed); against the supervised baseline it nearly vanishes on seed 81 (0.805 against 0.803).

The chapter does not promote this into robustness: the noise is still Gaussian, the shift is one scalar, and the gaps are within seed noise of each other. It is reported as a bounded degradation curve with a consistent sign, not as proof that relational supervision hardens signal classifiers. What the curve does rule out is the fragile alternative: if B’s relations were a brittle overfit to clean signals, doubled noise should hurt B more than A — instead B degrades the least on every seed. Consistently least, narrowly, and within noise: three qualifiers the sentence needs, and the sentence keeps all three.

Example C — the collapse that did not happen

Representational collapse would show as embedding spread near zero: every signal mapped to one point, relations trivially matched, accuracy at chance. The spread diagnostic (mean per-dim standard deviation of test embeddings) reads 1.6–1.8 in all arms on all seeds — two orders of magnitude above collapse, and indistinguishable between arms.

The diagnostic earns its place by ruling out the degenerate reading of B’s low RKD: the relations match without the embeddings having collapsed, which is consistent with structure having transferred — and with spreads of 1.6–1.8 in every arm (C slightly lower on every seed), no arm bought its RKD value with a collapsed or exploded scale either. A contrastive objective without normalisation, or an RKD weight an order of magnitude larger, might have collapsed; at \(\lambda = 0.5\) with normalised vectors, nothing did. The negative is the finding’s guardrail, and it is measured rather than assumed.

The frozen E2 extension

Seed Arm Acc Match RKD Spread NoisyAcc
80 A-signal 0.985 0.978 0.081 1.698 0.780
80 B-relational 0.990 0.978 0.053 1.729 0.790
80 C-shuffled 0.985 0.973 0.073 1.613 0.777
81 A-signal 0.993 0.990 0.040 1.833 0.803
81 B-relational 0.988 0.990 0.038 1.831 0.805
81 C-shuffled 0.988 0.993 0.041 1.765 0.795
82 A-signal 0.995 0.983 0.053 1.664 0.780
82 B-relational 0.995 0.988 0.046 1.691 0.803
82 C-shuffled 0.993 0.985 0.052 1.603 0.785

Aggregates: accuracy 0.991/0.991/0.988; matching 0.983/0.985/0.983; RKD 0.058/0.046/0.055; noisy 0.787/0.799/0.786. Teacher 1.000 on validation and test, all seeds — the gate passed with headroom, so no arm can blame the source. Agreement tracks accuracy (teacher always right). All arms ran the full 40-epoch budget; students cost ~0.2 s (A) and ~1.2 s (B/C) per seed against the teacher’s ~1.6 s; inference is ~0.004 ms per query.

Three readings. H1 holds: the image teacher is a competent source. H2 holds weakly and consistently: B’s held-out RKD is the lowest on all three seeds, matching ties-or-leads, noisy accuracy leads by ~0.01, accuracy ties — relations without accuracy movement. H3 holds: C’s RKD exceeds B’s on all three seeds while accuracy ties everywhere — destroying the correspondence removes the relational advantage (and B’s small noisy-accuracy edge with it) while leaving accuracy untouched. The chapter refuses the two upgrades the table tempts: matching at 0.98 is not cross-modal understanding (a linear map plus class-separated clouds will do that), and RKD at 0.046 is not a capability (no downstream task consumes it here).

Ordered relations, tied accuracy

Figure 2. Accuracy ties near ceiling while relations separate the arms: B lowest on all seeds, C above B on all seeds. Frozen E2 extension measurements.

What the metric cannot see

Teacher–student agreement is reported (0.985–0.995) and dismissed in one sentence: with a perfect teacher it equals accuracy, so it adds no information and is never cited as transfer evidence. The honest transfer metrics are matching (layout agreement through a fitted map) and held-out RKD (relation agreement with no fitted map) — and both need their caveats stated. Matching fits a 16→32 linear map on training embeddings; a sufficiently expressive map can manufacture agreement from any two class-separated spaces, so matching near 1.0 is a weak test that B passes and C nearly passes (0.983).

RKD has no fitted map, which makes B lowest (with C above B) the stronger finding — and it is also the ordering most vulnerable to the saturated-accuracy critique: the relations differ, but would any downstream decision change? On this task, no measured one would. Chapter 24’s weak, seed-concentrated retrieval gain is the direct precedent for this discipline: report the representation movement, refuse to convert it into a capability claim, and keep the null result (here: accuracy) visible beside it.

Cost, preprocessing, and rights

Teacher training costs ~1.6 seconds per seed (40 epochs on 600 images); students cost ~0.2 s (A) and ~1.2 s (B/C, which pay teacher forward passes for relations). All arms ran the full 40-epoch budget; unique exposures are the full splits. Total study wall time 13.8 seconds with a 493.5 MB peak process high-water mark.

Inference is one forward pass per query in every arm (~0.004 ms); relational supervision costs nothing at test time — the relations were a training signal, and the deployed student is the same 1,140 parameters in all arms. Preprocessing is part of the ledger: signals standardised with train statistics, images clamped, pairings fixed by seed. Change any of them and the numbers change; the manifest records all three.

The study uses RM-121, VERIFIED_PERMITTED for train_multimodal_distillation_e2. The corpus is self-created in both modalities with construction-recorded labels; no human images, voices, biometric data, or personal data exist anywhere in the study — the audio modality is a prototype-plus-noise vector, and the chapter states that this is why no speech conclusions are drawn. The legal context is background, not advice: GDPR special-category and minimisation duties (L-021) are why biometric data would need a recorded basis and why this study uses none; AI Act provider and deployment duties (L-023) are the regime a real multimodal system would face and this synthetic study does not trigger. RM-104 and all prior records are untouched.

Limitations

One synthetic pairing, three seeds, tiny models. Noise is Gaussian at two levels; the robustness curve has exactly two points. Accuracy is saturated near 0.99, so the comparison lives in relations, matching, and robustness — a strong representation result would need headroom this task does not provide. No linear probe (centroid matching plus held-out RKD serve instead; the omission is recorded, not hidden). No temporal signals, no real speech, no images of people. Speech and audio claims are background from DistilHuBERT (P-052); nothing about real speech is measured here. No external replication and no independent review.

Preparing for fewer steps

Two senses agreed — approximately — on one label set. Chapter 30 asks what happens when the output is not a label at all but a sample from a distribution, and the teacher’s advantage is not knowledge but compute: dozens of denoising steps the student cannot afford. The carrier changes again, from relations between examples to trajectories through noise, and the evaluation changes with it: not accuracy but fidelity, not agreement but coverage.

The discipline does not change: an independent check, a cost ledger, and no credit for matching the teacher where the task does not follow. And one question carries over intact: when the student operates with fewer senses — or fewer steps — than the teacher, which differences are the method working and which are the evaluation flattering it.

References

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G. and Sutskever, I. (2021). Learning Transferable Visual Models From Natural Language Supervision. Proceedings of the 38th International Conference on Machine Learning (ICML). arXiv:2103.00020. https://arxiv.org/abs/2103.00020

Chang, H.-J., Yang, S.-w. and Lee, H.-y. (2022). DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). arXiv:2110.01900. https://arxiv.org/abs/2110.01900

Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). Applicable 25 May 2018. https://eur-lex.europa.eu/eli/reg/2016/679/oj (retrieved 12 November 2026). Cited as L-021 for the special-category, minimisation, and lawful-basis context: biometric and voice data would need a recorded basis; this study uses none. Not legal advice.

Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). In force 1 August 2024. https://eur-lex.europa.eu/eli/reg/2024/1689/oj (retrieved 12 November 2026). Cited as L-023 for the provider and deployment-obligation context that a real multimodal system would face and this synthetic study does not trigger. Not legal advice.