Multi-Agent and Expert Distillation
Can two specialised teachers transfer complementary capabilities to one student? Scripted specialists, a learned router, oracle-routing diagnostics, and a bounded E4 study where routing succeeds everywhere and trajectory execution almost nowhere.
Chapters 26 and 27 distilled one teacher — scripted, optimal, alone. Real deployments rarely look like that. Knowledge lives in several places: one model that handles one region well, another that handles the rest, a router that decides who answers. The question is what “distilling” those teachers means, and what gets lost when several behaviours are aggregated into one student:
Can two specialised teachers transfer their complementary capabilities to one student, and what is lost when their behaviour is aggregated?
The measured answer separates three competences that a single success number would fuse. Teacher competence: each specialist succeeds on its own half, verified by the gate that would have blocked the study otherwise. Routing competence: a learned router assigns all 36 held-out queries to the right specialist. Student trajectory competence: 2 successes in 108 student episodes. The router works; the oracle works; the students do not. A positive routing result without successful episodes is not transfer, and the experiment exists to make that distinction checkable.
Figure 1. Three competences, measured separately: routing 1.000, diagnostic oracle 1.000, students 0.056/0.000/0.000. Frozen E4 measurements (36 held-out queries).
Three things called “many teachers”
The contract requires the distinction first, because the literature uses one vocabulary for three systems. Multi-teacher distillation (Chapter 11) trains one student from several teachers’ outputs on the same task — the teachers disagree about how, not about what. A mixture of experts keeps the experts at inference time and learns a gating network that routes each input to a subset; the “student” is really a routing system plus specialists, and distillation enters when the mixture is compressed or when routing itself is learned.
Multi-agent distillation would transfer behaviour from agents that act, observe, and coordinate — trajectories with interaction, not just labels with disagreement. Our study is the middle case with the routing made explicit: two specialists, one router, students that either absorb the mixture into weights or observe the routing decision. No collaboration or debate traces exist here; the chapter records that as a boundary rather than borrowing a multi-agent label for a routing study.
Specialists with a genuine boundary
Specialist T0 owns entities 0–7; T1 owns 8–15. On its own half, a specialist executes the optimal lookup–read–answer policy. Off its half, it answers 0 immediately and usually fails. The off-half exit is deliberately crude: it makes the specialisation boundary measurable. Two teachers that succeed everywhere are not specialists; two teachers whose first actions agree nowhere (measured first-action agreement: 0.000 on all seeds) and whose final answers agree only where the truth happens to be 0 (0.083) are complementary by construction, and the disagreement numbers certify it. A positive control with teeth: had either specialist scored below 1.000 on its own-half validation queries, the dependent arms would have been blocked honestly. Both passed on all seeds.
Example A. Query (2, 1): value \((10+7)\bmod 16 = 1\). T0 rolls out LOOKUP(2), READ(2,1), ANSWER(1) — success. T1 answers 0 at once — failure. Query (9, 2): value \((45+14)\bmod 16 = 11\). T1 rolls out the three steps — success. T0 answers 0 — failure. The same query produces a success trajectory and a one-step failure depending on who is asked. Any student trained on both teachers’ full behaviour must somehow not average these into answering 0.5 — the failure mode the aggregation arms test.
Example A′. The disagreement trace for query (2, 1), both teachers side by side. T0: \(o_0\) → LOOKUP(2) → \(o_1\) → READ(2,1) → \(o_2\) (value 1 observed) → ANSWER(1) → success in 3 steps. T1: \(o_0\) → ANSWER(0) → failure in 1 step. First actions differ (LOOKUP vs ANSWER); final answers differ (1 vs 0); step counts differ (3 vs 1). This is the required multi-agent trajectory with disagreement — two rollouts, one query, no consensus — and the audit table compresses 12 such pairs per seed into first-action agreement 0.000 and final-answer agreement 0.083. A student that imitates “the teachers” without an ownership map imitates this contradiction.
Gating, load balancing, and why routing collapses
A mixture-of-experts layer (Shazeer et al., P-050) computes, for input \(x\), gate values \(g_i(x) = \mathrm{softmax}(W_g x)_i\) over experts \(E_i\) and outputs \(y = \sum_i g_i(x) E_i(x)\), usually sparsified to the top-\(k\) gates so most experts sit out each example. Capacity grows with the expert count while per-example compute stays near the cost of \(k\) experts — the paper’s headline result, reported with thousand-fold capacity gains on language and translation tasks at modest extra compute, is theirs, not ours, and the chapter borrows only the mechanism.
The gated mixture is a conditional distribution in its own right. With a hard top-1 gate, \(p(y \mid x) = p_{E_{r(x)}}(y \mid x)\) where \(r(x) = \arg\max_i g_i(x)\) is the routed expert; with soft weights, \(p(y \mid x) = \sum_i g_i(x)\, p_{E_i}(y \mid x)\), a mixture whose components are the experts. Distilling the mixture can therefore mean two different things — matching the mixture distribution \(\sum_i g_i p_{E_i}\) (ensemble-style, Chapter 11) or matching the routing rule \(r(x)\) plus each expert conditionally — and the two have different failure modes.
Matching the mixture averages away the ownership information: a student that reproduces \(\sum_i g_i p_{E_i}\) need never know which expert owned an input. Matching the router preserves it, at the price of executing the (possibly wrong) routing at test time. Arms B and C are these two choices in miniature: B sees no ownership signal, C observes it in training and predicts it at test.
The load-balancing problem follows from the objective: the gate is trained to send each example where it is best served, but “best served” is observed only for the experts actually used, so a gate that initially prefers expert 0 trains expert 0 more, which justifies preferring expert 0 — the rich get richer until one expert serves everything. Shazeer et al. counter with auxiliary losses computed per batch: with importance \(I_i = \sum_{x} g_i(x)\) (total gate mass per expert) the importance loss \(L_{\mathrm{importance}} = \mathrm{CV}(I)^2\) penalises uneven mass, and a load loss penalises uneven fractions of examples routed to each expert, where \(\mathrm{CV}\) is the coefficient of variation (standard deviation over mean).
A worked two-expert batch makes the scale concrete: gate masses \((0.9, 0.1)\) have mean 0.5, standard deviation 0.4, so \(\mathrm{CV} = 0.8\) and the loss is \(0.64\); even masses give loss \(0\). Our router has no such loss — it trains on plain cross-entropy against the owning specialist — and the utilisation audit is the check that collapse did not happen anyway: routed splits of 0.42/0.58, 0.50/0.50, 0.58/0.42 with entropies 0.68–0.69 nats against a maximum of \(\ln 2 \approx 0.693\). Balanced, and the balance is measured, not assumed. For the seed-90 split (0.42/0.58): \(H = -0.42\ln0.42 - 0.58\ln0.58 \approx 0.365 + 0.316 = 0.681\) nats — within 0.02 of maximum, the quantitative form of “no expert starved”.
Example B. The collapse fixture the test suite pins: a router trained only on half-0 queries predicts specialist 0 for everything — utilisation 1.00/0.00, entropy 0. A deployment that monitored accuracy alone would call it perfect (on half-0 queries, it is); the utilisation audit calls it collapsed. Report the routed split with every routing accuracy, or the accuracy number is uninterpreted.
Ensembling outputs differs from learning a router in the object being transferred. An ensemble averages predictions (Chapter 11): \(\bar p = \sum_i w_i p_i\) with fixed or fitted weights. A router transfers a decision rule: which expert owns this input. The ensemble can be right while every expert is wrong in different ways; the router is right only when the ownership map is right. The distinction bites hardest on discrete actions, where averaging is not even defined: T0 says LOOKUP(2), T1 says ANSWER(0), and their “average” is not a tool call.
Any multi-teacher method that blends action distributions must either pick (routing), vote (majority), or sequence (debate) — averaging logits and taking the argmax is a fourth option that silently lets the larger-logit teacher win every disagreement, which is routing with the gate hidden inside the temperature. Our arm B is ensemble-like (one policy from both teachers’ correct demos, no ownership signal); arm C is router-like (ownership as an observed input in training, a learned prediction at test). Both score 0.000 on episodes — the comparison the study can make is not which wins but how each preserves, or fails to preserve, the ownership information. C preserves it measurably (routing 1.000); B has no slot for it. Neither executes.
A student that internally routes between experts differs from a single student that absorbed their behaviour in exactly this slot: the routed student carries an explicit ownership representation (our conditioned policy’s two extra input dimensions, set by the router at test time), while the absorbed student must re-derive ownership from the observation on every step.
The experiment measures the value of the slot: with oracle routing C still scores 0.000, so the slot is not sufficient — knowing who owns the query does not teach the policy how to act. That is the sense in which routing and execution are separate competences, and the sense in which a mixture with a perfect gate can still fail as a distilled student.
The frozen E4 extension
Forty train queries per seed (90, 91, 92); twelve held out for test, twelve for validation reference. Four students at EXACTLY 2,000 updates each: A on T0’s demos everywhere including T0’s off-half exits (82 unique pairs); B on correct-specialist demos (120 pairs, no routing signal); C a router (2,000 updates on 40 query→specialist pairs) plus a conditioned policy trained with oracle specialist ids and evaluated with learned routing; D oracle routing executing the correct specialist’s action, untrained, diagnostic only.
| Seed | A | B | C | D | Router | Util T0/T1 |
|---|---|---|---|---|---|---|
| 90 | 0.083 | 0.000 | 0.000 | 1.000 | 1.000 | 0.42/0.58 |
| 91 | 0.083 | 0.000 | 0.000 | 1.000 | 1.000 | 0.50/0.50 |
| 92 | 0.000 | 0.000 | 0.000 | 1.000 | 1.000 | 0.58/0.42 |
Aggregates: A 0.056 ± 0.048, B/C 0.000, D 1.000, router 1.000 on all seeds. C with oracle routing also scores 0.000 on all seeds — conditioning on the right specialist does not save execution, so the routing→execution gap is not an artefact of router error. Success by half for A: 0.000 on half-0 everywhere, 0.143/0.167/0.000 on half-1 — the only student successes in 108 episodes, all on T0’s failing half, consistent with memorising T0’s ANSWER(0) exit and landing on queries whose truth is 0.
Imitation of failure that happens to be right: the chapter labels this an interpretation of two episodes, not a finding, and draws no ranking from it. What each arm actually trained on matters more than the headline means: A saw 82 pairs including T0’s off-half exits (failures as demonstrations), B saw 120 all-correct pairs with no ownership signal, C saw the same 120 with ownership labels plus a 40-pair router. Same update budget (2,000), same optics, different supervision — and the same floor. The supervision differences the chapter can measure therefore live entirely in the routing channel (1.000) and the oracle channel (1.000), not in the executed episodes.
Four readings. H1 holds: the specialists pass their gates (1.000 own-half, or the arms would have been blocked). H2 is the expected null: students do not rise above the Chapter 26 floor. H3 holds and is the finding: routing accuracy 1.000 with C episode success 0.000 on the same 36 queries — the router learned the ownership map while the policy learned no executable behaviour. H4 holds: D succeeds 12/12 on every seed, so the task is solvable and the failure is the student’s trajectory competence, not the teachers’, the router’s, or the task’s. D does so by construction in exactly 3 scripted steps per episode with zero invalid calls — no learning, no search, just the correct specialist acting.
The diagnostic value is the contrast it sets up: the same three steps that D executes flawlessly are the three steps every learned policy fails to reproduce, which locates the deficit precisely in trajectory execution rather than in task difficulty, routing, or teacher quality. Per successful episode, D spends 3 scripted tool calls; A spends 3 learned forwards for 0.056 success; B and C spend 3–4 learned forwards for 0.000. The cheapest success in the study is also the least learned — a fact the chapter reports without resolving, since resolving it (can execution be distilled at all here?) is the open question the floor result poses.
Teacher disagreement, measured
The disagreement audit runs T0 and T1 independently on each test query and compares. First-action agreement is 0.000 on all seeds: where one looks up, the other answers 0 — structural, by design, and the number that certifies the specialisation is real rather than nominal. Final-answer agreement is 0.083: the teachers agree only where the truth is 0 and both say 0 — one query in twelve per seed, the same rate at which blind ANSWER(0) scores. For distillation, the audit answers the question the aggregation arms cannot: the teachers furnish no consensus to average. Where the two final answers coincide, both say 0; everywhere else the correct teacher’s answer is the truth and the other’s is 0.
An aggregator that averaged final answers would be wrong everywhere except the 0-truth queries — which is to say, averaging is refuted as a strategy before any student trains, by arithmetic on the audit table, not by a failed run. Arm B’s training data contains no contradiction (each demo comes from the correct specialist), which is precisely why B cannot be accused of “averaging away disagreement” — its failure is execution, the same floor as Chapter 26, on fully consistent data. Arm A’s data does contain contradiction (T0’s failures alongside its successes), and A’s two successes come from the failure side. Negative transfer here is not degradation relative to a working baseline; it is the absence any baseline above zero would be needed to show. The chapter reports the null rather than manufacturing a ranking from two episodes.
Costs, utilisation, and collapse monitoring
Policy training costs ~0.9–1.0 seconds per arm per seed (2,000 updates on 82–120 pairs); the router adds ~0.8 seconds per seed; D costs nothing. Unique exposures differ by design (82/120/120) with matched update budgets. Total study wall time is 11.6 seconds with a 455.2 MB peak process high-water mark. Inference per step: A/B one 10,400-parameter policy forward; C one 370-parameter router plus one 10,528-parameter policy forward; D scripted calls with no learned forwards. Per successful episode the accounts diverge sharply: D spends 3 scripted tool calls for 1.0 success; A spends 3 learned forwards per episode for 0.056; B and C spend 3–4 learned forwards per episode for 0.000.
The coordination overhead of routing is therefore one small forward pass per decision — negligible here, and the chapter states plainly that at this scale the overhead cannot discriminate designs; what discriminates them is utilisation and collapse behaviour, both healthy (entropy 0.68–0.69 nats, no expert starved). A routed system that collapses pays the router’s cost for a single expert’s performance; ours does not collapse, so the comparison is honest. The generalist baseline (arm A: T0’s demos everywhere, including its off-half exits) is the ablation the contract requires: a single student from a single teacher, failures included, against which the combined and routed students are read — and all three meet at the floor, so the ablation reports the floor three ways rather than ranking it.
Expert utilisation is reported as the routed test split with its entropy, not as a prose adjective. “Balanced” means 0.42–0.58 with entropy within 0.02 nats of maximum; the collapse fixture means 1.00/0.00 with entropy 0. Any future expert addition that moves these numbers is a regression with a name, not a vibe.
Rights: whose terms does the student inherit?
Each expert may carry its own model terms, data licence, and attribution requirement, and the student may inherit the strictest combination — a restriction is not diluted by mixing. The register records the two archetypal cases: Gemma’s terms expressly treat distillation using intermediate representations or synthetic outputs as a regulated Model Derivative with distribution conditions (L-004), and the Llama community licence attaches naming, notice, acceptable-use, and scale conditions to derivative use (L-005). Both teachers here are self-created scripts on synthetic data, so both records describe a regime with nothing to inherit — RM-120 names T0 and T1 individually and records that the student inherits no broader rights than the record grants.
The section exists so the next multi-expert study, with real experts, has the question pre-asked: list every expert, record every term, reject incompatible combinations before training. Concretely, a Gemma-licensed teacher mixed with an Apache-licensed teacher yields a student governed by the stricter set — the Gemma distribution conditions do not wash out in the mixture, and an “average” of the two licences is not a licence at all. Unknown remains unknown; it is never permission.
Failure modes, honestly listed
Routing collapse: not observed (utilisation balanced); the fixture shows what it would look like. Incompatible licences: not present (both experts self-created); the L-004/L-005 context shows what to check. Averaging away disagreement: not tested directly — B trains on contradiction-free demos and still floors, which shows contradiction is not needed for the failure here. Coordination cost exceeding benefit: measured as one extra forward per step for zero episode gain — the cost is real, the benefit absent, and the chapter does not hide the ratio. Calling routing a multi-agent method: refused — no agent interacts with another, no trajectory contains another agent’s actions, and Chapter 28 stays a routing study so Chapter 29’s modality gap and a future collaboration study have clean boundaries.
The utilisation audit has a resolution limit worth stating: it is measured per half (entity < 8 vs ≥ 8), so a subtler collapse that serves a half’s entities while neglecting its rare attributes would pass unnoticed — collapse monitoring is only as fine as its partition, and attr-level neglect is unmeasured. For the forward links: Chapter 11’s ensemble machinery is the averaging alternative C was built to differ from; Chapter 26’s hypothesised memorisation floor is the mechanism H2 invokes; Chapter 27’s store is the ownership representation done right (a key to delete beats a weight to retrain); Chapter 35 will take routing collapse as a safety regression when the router guards a capability boundary rather than an entity half.
Limitations
Two scripted specialists; no learned experts, no collaboration or debate traces. The episode floor is expected from the mechanism hypothesised in Chapter 26 (D-FP-12B), and with two successes in 108 episodes no student ranking is claimed. The router is evaluated on a seen-entity distribution; unseen-entity routing is unmeasured — and it is the measurement that would matter most, because memorising “entities 0–7 go left” from a training set that names every entity is consistent with both a rule and a lookup table, and only an unseen entity tells them apart.
The study’s forty training queries cover all sixteen entities on every seed (checked post hoc: no seed missed an entity), so the router was never forced to generalise to a new owner, only to new combinations of known owners and attributes. Routing entropy near maximum is a health check, not proof against subtler collapse (e.g., attr-level neglect — unmeasured). No external replication and no independent review.
Preparing for new senses
Two teachers, one sense each would be the next complication: what changes when the experts do not even share an observation space — when one reads images and the other reads signals? Chapter 29 asks exactly that. The routing question becomes an alignment question: before a student can choose between modalities, let alone distil from one to another, the representations must meet somewhere.
The router’s ownership map assumed a shared query encoding — the same twenty-dimensional entity/attribute vector for router, policy, and both specialists — so routing never had to solve correspondence. Cross-modal transfer has to build the shared space first, from paired data whose two sides show the same underlying fact in different senses, and the chapter shows what survives the construction: not the teachers’ coordinates, which are incommensurable across senses, but the relations between examples, which can match even when the spaces do not.
References
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. V., Hinton, G. and Dean, J. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. International Conference on Learning Representations (ICLR). arXiv:1701.06538. https://arxiv.org/abs/1701.06538
Legal sources
Google Gemma Terms of Use. Last modified 1 April 2026. https://ai.google.dev/gemma/terms (retrieved 12 November 2026). Cited as L-004 for the Model Derivative and distribution-conditions context for multi-source distillation; not legal advice.
Llama 3.2 Community License Agreement. Version released 25 September 2024. https://www.llama.com/llama3_2/license/ (retrieved 12 November 2026). Cited as L-005 for the naming, notice, acceptable-use, and scale conditions context for derivative use; not legal advice.