← Distillation From First Principles

Tool Use and Agent Trajectories

Can a student learn to complete multi-step tasks by distilling a teacher's tool-use trajectories? Agent policies, behaviour cloning, compounding errors, distribution shift, recovery probes, and a bounded E4 study where cloning memorises but never generalises.

Chapter 25 generated an output and checked it. The whole task was one decision. An agent faces a different structure: it chooses a tool, reads the observation, chooses again, and only after several decisions produces anything checkable. Each decision changes what the next decision sees:

Can a student learn to complete multi-step tasks by distilling a teacher’s actions and tool-use trajectories?

On our bounded study, the measured answer is no — and the way it fails is the chapter’s real subject. Three policies, three seeds, 108 held-out episodes: zero successes. Yet the same policies agree with the teacher on roughly 30% of held-out teacher states, and the recovery-augmented student answers a third of forced error states correctly while the plain cloning student answers none. Imitation without generalisation, local robustness without task success, agreement without completion. The trajectory experiment exists to keep those apart.

Agent policies

An agent policy maps observations to actions, \(\pi_\theta(a \mid o)\). The observation \(o\) encodes everything the agent may condition on: the original query, the last tool’s result, how many steps have been used. The action \(a\) is a discrete tool call: which tool, with which arguments — or the final answer, which ends the episode. A teacher trajectory is a sequence \(\tau = (o_0, a_0, o_1, a_1, \ldots, o_H)\) produced by rolling out a teacher policy in the environment, where each observation follows deterministically from the previous observation and action.

Our environment is a synthetic lookup task, deliberately small enough to inspect completely. There are 16 entities and 4 attributes, giving 64 (entity, attribute) queries. The value for a query is \((5 \cdot \mathrm{entity} + 7 \cdot \mathrm{attr}) \bmod 16\). Three tools exist. LOOKUP(entity) records an entity as current and returns an acknowledgement.

READ(entity, attr) returns the value, but only if that entity is current — reading before looking up, or reading a different entity, is invalid and returns an error observation. ANSWER(value) ends the episode; the episode succeeds exactly when the value is correct. The step budget is 6. All tools are local deterministic functions. No network call exists, no external API is contacted, and no terms of service govern anything beyond the repository’s own licence record.

The observation is a 64-dimensional binary vector: one-hots for the query entity (16) and attribute (4), the last tool state (none, lookup, read, error), the last returned value (none plus 0–15), the steps used so far (0–5), and the currently looked-up entity (none plus 0–15). The current-entity field is load-bearing: without it, a state after a wrong lookup looks identical to a state after the right one, and recovery from the wrong lookup is unlearnable.

The implementation learned this the embarrassing way — the first version omitted the field, and the recovery fixture it was built to support could not work. The field was added before any valid run, and the protocol records the environment as version e4-lookup-v1 so the mistake is not silently repaired.

Example A. Query: entity 5, attribute 2. The teacher’s trajectory: \(o_0\) (query 5/2, nothing done) → LOOKUP(5) → \(o_1\) (entity 5 current) → READ(5,2) → \(o_2\) (value 7 observed, since \((25+14)\bmod 16 = 7\)) → ANSWER(7) → success in 3 steps. Every teacher trajectory has this shape: three actions, zero invalid calls, success by construction. There is no failed teacher trajectory in this study — a limitation the chapter returns to, because learning only from successes may contribute to the floor result.

Example B. A failed student rollout on the same query, from the frozen run: \(o_0\) → LOOKUP(5) (correct) → \(o_1\) → READ(5,0) (wrong attribute — the student memorised a training read for entity 5) → \(o_2\) (value 9 observed, wrong) → ANSWER(9) → failure in 3 steps. No invalid call occurred; every action was legal and every observation well-formed. The student did not crash. It confidently executed the wrong plan. Invalid-call rate and episode success measure different things, and this rollout is why both are reported.

Same start, different fate at step two

Figure 1. Examples A and B side by side: identical first two steps, divergence at READ, success against confident failure. Frozen-run walkthrough (query 5/2).

The behaviour-cloning objective

Given a dataset \(\mathcal{D}\) of teacher state-action pairs \((o, a^*)\), behaviour cloning minimises

\[ \mathcal{L}_{\mathrm{BC}}(\theta) = -\mathbb{E}_{(o,a^*)\sim\mathcal{D}} [\log \pi_\theta(a^* \mid o)]. \]
For a discrete action space this is cross-entropy with the teacher's action as the hard target — the same mathematics as supervised fine-tuning, applied to states instead of prompts. Its gradient, \(-\nabla_\theta \log \pi_\theta(a^* \mid o)\), raises the probability of the demonstrated action at the demonstrated state and says nothing about any other state. That last clause is the whole story of this chapter: the objective supervises 120 training states (40 queries × 3 steps) and is silent about the infinitely many others, including every state the student itself will visit.

The trajectory-likelihood view makes the silence explicit. Under a policy \(\pi\), a trajectory’s probability factorises as

\[ P_\pi(\tau) = \prod_{t=0}^{H-1} \pi(a_t \mid o_t) \cdot T(o_{t+1} \mid o_t, a_t), \]
where \(T\) is the environment transition (deterministic here). Behaviour cloning maximises the likelihood of the teacher's actions at the teacher's states — the \(\pi\) factors at visited \(o_t\) — and never evaluates \(\pi\) at states the teacher never visited. At test time the student generates its own states, and the first action that differs from the teacher's puts it somewhere the objective never priced. From there it acts off-distribution for the rest of the episode.

Compounding turns a per-step weakness into episode failure. Suppose the student matches the teacher with probability \(1-\epsilon\) at each step independently. Episode success needs all \(H\) steps right, so success is \((1-\epsilon)^H\). With the measured per-step disagreement near 0.7 (\(\epsilon \approx 0.7\)) over \(H=3\) steps, expected success is \(0.3^3 \approx 0.027\) — about one episode in 36. Observed: zero in 36. The arithmetic does not prove independence (the errors are correlated, not independent draws), but it sets the scale: a 30%-agreement policy is not “almost working”; over three steps it is a zero-success policy with probability approaching one. Report per-step accuracy for a multi-step agent without the horizon calculation and you will mistake the floor for the staircase.

The horizon is the amplifier

Figure 2. A 30%-agreement policy over three steps: expected 0.027 episodes per batch, observed zero in 36 rollouts. Frozen E4 measurement; independence disclaimed in prose.

Distribution shift, concretely

Distribution shift in this chapter is not a metaphor for “different data”. It is the gap between two precisely defined distributions over the same observation space: the teacher-state distribution (states visited by rolling out the teacher) and the student-state distribution (states visited by rolling out the student). Training minimises loss on the first; evaluation runs on the second. The moment the student’s first action differs — and with 0.3 agreement it usually does at step one — every subsequent observation is drawn from a distribution the objective never saw.

The frozen run measures both sides. Per-step agreement is computed on held-out teacher states: fresh queries, but the teacher’s states, not the student’s. Episode success is computed from the student’s own rollouts. Arm B agrees on 0.278–0.361 of teacher states and completes 0 of 36 episodes. The two numbers describe the same policy on two distributions, and their divergence is the shift made visible. This is why the chapter insists on both metrics: agreement alone would report “the student learned 30% of the behaviour”, while the rollouts show it completes nothing. Neither number is wrong; each answers a different question, and only together do they describe a policy.

There is a second shift worth naming because it limits what the study can claim. The recovery states in arm C are synthetic fixtures — wrong-lookup and failed-read states built by hand with known correct recoveries — not states any policy actually visited. They teach the correct action at two error states per query. The recovery probe then measures exactly those states. C’s probe advantage (0.361 against B’s 0.000) therefore shows that declared error states can be taught, not that naturally occurring errors are handled. The chapter does not upgrade the fixture into a finding about robustness in the wild.

The limits of successful demonstrations

Every teacher trajectory in this study succeeds. That is a design choice with a cost, and the cost is selection bias: the student never sees what failure looks like, which states precede it, or which actions recover. Three consequences follow.

First, the student cannot learn error avoidance. Invalid calls are absent from training (the teacher makes none), so the student’s invalid-call behaviour at test is untrained extrapolation. Measured: near-zero invalid rates (0.009 overall, all from one seed) — the students rarely call tools illegally, not because they learned the constraint but, on the memorisation hypothesis, because their memorised actions happen to be legal ones. A constraint that is never violated in training is not a constraint the student understands.

Second, the student cannot learn recovery. Arm C’s synthetic fixtures are the admission ticket for this lesson: without them, the recovery probe sits at exactly 0.000 on all three seeds. A policy trained only on successes answers every error state wrongly. The probe result is the chapter’s one positive transfer — declared states, taught responses — and its narrowness is the point. Two hand-built error states per query transfer; the space of real errors remains uncovered.

Third, success-only data hides the teacher’s competence boundary. The scripted teacher succeeds everywhere, so the data cannot tell the student which queries are hard, which tool sequences are fragile, or where the teacher itself would fail. A learned teacher’s failures would be informative; a scripted teacher’s perfection is uninformative. The chapter records this as the reason the next study needs either a fallible teacher or explicitly curated failure coverage — and as a warning against reading “trained on expert trajectories” as “trained on the full task”.

The gradient makes the bias precise. Behaviour cloning’s update, \(-\nabla_\theta \log \pi_\theta(a^* \mid o)\), moves mass toward the demonstrated action with no term for the actions not taken and no weight for whether the demonstration was good — every teacher action is treated as equally worth imitating, including the ones that only look optimal because the failures were filtered out before training. Chapter 25 observed a 3% demonstration noise alongside a five-point gap on one seed; here the bias is structural rather than statistical.

With zero failures in the data, the objective’s implicit claim is that failure states do not exist, and the student’s first encounter with one — B’s probe score of exactly 0.000 on all 72 forced error states — is the price of that claim. What a failed teacher trajectory would teach is therefore specific: not just “this action is wrong” but the state that precedes the wrong action and the recovery that follows it. Arm C’s fixtures are a hand-built substitute for two such states per query; the probe gains (+0.333, +0.292, +0.458) measure the substitute, not the thing itself.

Where this sits in the literature

Two papers frame the design space without supplying the method. ReAct (Yao et al., P-032) interleaves reasoning traces with actions, so the model’s “thoughts” induce, track, and update its action plans and handle exceptions: acting interfaces with external sources while reasoning interprets the results. Our trajectories are act-only — observations and tool calls, no reasoning traces — which makes them a strict subset of the ReAct format and a deliberate one: the chapter measures whether action sequences alone transfer, without the reasoning scaffold.

Toolformer (Schick et al., P-033) takes the opposite route to supervision: instead of imitating a teacher, it self-supervises API calls by keeping the ones that reduce language-modelling loss, requiring only a handful of demonstrations per API. Our supervision is the mirror image — a scripted teacher’s demonstrations with no loss-based filtering — and Toolformer’s open questions (which APIs, with what permissions, under what execution safety) are exactly what our simulator answers by avoidance. Neither paper is a distillation study, and the chapter borrows no result from either; both are cited for the trajectory and tool-use framing the experiment narrows to a measurable instance.

Invalid calls, recovery, and safe simulation

An invalid call in this environment — reading an entity that is not current — returns an error observation and consumes a step. It does not raise, crash, or corrupt state. That gentleness is a design decision, not realism: real tools fail louder (authentication errors, rate limits, destructive writes). The error observation exists so recovery can be measured at all. The recovery probe then asks the policy, at 24 forced error states per seed, for the correct next action.

The frozen rollouts add one opportunistic data point: on seed 60, one of twelve episodes in each tool arm contained an invalid call, and that episode failed in both arms (recovery_given_invalid 0.0; seeds 61 and 62 produced no invalid episodes at all). A single episode proves nothing, and the chapter does not promote it beyond what it is — but its direction matches the probe: naturally occurring recovery, untaught, does not happen here.

All tool execution is simulated in the most literal sense: the tools are pure functions of their arguments, the knowledge base is a modular-arithmetic rule, and no secret, personal record, or external system is reachable. “Safe simulated tool execution” here means there is nothing unsafe to execute — no shell, no file writes, no network. A chapter that distils browser or shell tool use would need sandboxing, capability restrictions, and audit logging; this one documents why its task needs none of those, and states plainly that the safety argument ends at the simulator boundary.

Redaction is demonstrated, not needed. Trajectories in this study contain integers, never names, credentials, or personal data. The redact_secrets utility scans string fields of trajectory records for declared secret substrings and replaces them, returning the redacted records and a replacement count. The test suite pins it on a synthetic fixture (sk-live-42 → [REDACTED], count 1). The chapter includes it because the contract requires a redaction example, and labels it as a utility demonstration on synthetic data: it proves the function runs, not that any real secret was protected. Real trajectory hygiene — consent, minimisation, retention windows, deletion propagation — belongs to Chapters 27, 38, and 39, where the data is not integers.

The frozen E4 comparison

Sixty-four query combinations; per seed, 40 for training, 12 for validation reference, 12 held out for test. The teacher needs no training and succeeds everywhere by construction. Three policies per seed, each trained EXACTLY 2,000 optimiser updates with seed-matched inits where architectures match (B and C asserted identical at initialisation): A, a direct-answer MLP (1,200 parameters) on 40 query-value pairs; B, a tool policy (10,400 parameters) on 120 teacher pairs; C, the same tool policy on 120 teacher pairs plus 80 synthetic recovery pairs. Checkpoint by best training loss — no held-out selection, disclosed as a limitation.

Seed Arm Success StepAgree RecProbe Invalid Steps
60 A 0.000 n/a n/a 0.000 1.00
60 B 0.000 0.278 0.000 0.028 3.00
60 C 0.000 0.222 0.333 0.027 3.08
61 A 0.000 n/a n/a 0.000 1.00
61 B 0.000 0.278 0.000 0.000 3.00
61 C 0.000 0.250 0.292 0.000 3.00
62 A 0.000 n/a n/a 0.000 1.00
62 B 0.000 0.361 0.000 0.000 3.00
62 C 0.000 0.306 0.458 0.000 3.00

Aggregates: B step agreement 0.306 ± 0.048, C 0.259 ± 0.042; B probe 0.000 ± 0.000, C 0.361 ± 0.087; all episode successes 0.000 on all seeds, test and validation alike. Final-answer agreement equals episode success in every arm — the teacher is always correct, so matching its answer and succeeding are the same event, stated rather than hidden.

Three readings. H1 is null: trajectory cloning does not beat the direct-answer baseline, because neither completes anything. H2 is null on episodes and positive on the probe: C never completes an episode either, but it answers forced error states correctly a third of the time on every seed while B answers none. H3 is supported and is the finding: per-step agreement near 0.3 with rollout success at zero is compounding and distribution shift measured together. C’s slightly lower teacher-state agreement than B (0.259 vs 0.306) paired with its probe advantage is consistent with a capacity trade-off: fitting the recovery states may cost a little teacher-state fidelity and buy error-state coverage.

Why the floor is the finding

A table of zeros invites the suspicion of a bug. Three checks rule it out within the frozen evidence. First, the rollout machinery is verified independently: an oracle policy built from the same observation decoding completes episodes in 3 steps with zero invalid calls (pinned in the focused test suite). Second, the policies did learn something: step agreement of 0.28–0.36 is far above the 1-in-96 chance rate, and the recovery probe separates C from B on every seed — a broken loop would not produce that pattern. Third, the compounding arithmetic predicts it: \((0.3)^3 \approx 0.027\) expected episodes per 12 attempts is 0.3 episodes — zero is the modal outcome, not an anomaly.

The mechanism hypothesis, labelled as hypothesis: one-hot observation encodings with 120 training pairs let a 10k-parameter MLP memorise the training states (training agreement reaches 1.0 in a post-hoc check outside the frozen run) without extracting the copy rules — look up the query entity, read the query pair, answer the observed value — that would generalise to unseen combinations. The frozen evidence for the hypothesis is the shape of the failure itself: partial teacher-state agreement with zero rollout success is exactly what memorisation without rule extraction predicts. The chapter does not claim to have proved the mechanism; it claims the measurements, and offers the mechanism as the best current explanation.

When copying teacher output fails to transfer capability, this is the shape it takes in sequential tasks: not degraded performance but total failure, because a three-step task multiplies per-step gaps. Chapter 25’s cloning gap was five points on one seed; here the same phenomenon, compounded over three steps, sits at zero observed successes. The horizon acts as an amplifier, and any distillation claim for agent behaviour that reports only per-step accuracy is reporting the base without the exponent.

Cost and rights

Direct-answer policies cost ~0.9–1.9 seconds per seed (2,000 updates on 40 pairs); tool policies cost ~2.2–3.4 seconds (2,000 updates on 120/200 pairs). Unique exposures are 40/120/200 rows; examples seen are 64,000 per arm per seed (updates × batch, matched). Total study wall time is 21.4 seconds with a 456.0 MB peak process high-water mark. Rollout inference costs one policy forward per step — mean 3.0 steps for tool policies, 1 for the direct baseline. No API was called, no provider pricing exists, none is invented.

The study uses RM-118, VERIFIED_PERMITTED for train_agent_trajectories_e4. All tools are local deterministic functions; there are no external tool terms to accept and no tool version to drift. Trajectories carry query, teacher identity (scripted-optimal-v1 or synthetic-recovery-v1), and environment version on every pair. No personal data exists anywhere in the study; the redaction utility is demonstrated on synthetic secrets. RM-104 and all D-FP-11 records are untouched.

Limitations

One synthetic lookup environment, three seeds, tiny MLPs. The scripted teacher never fails, so success-only bias is total. Recovery states are declared fixtures, not observed errors. Selection is by best training loss, not held-out success. The step budget is 6; longer horizons are unmeasured. Episode success is zero everywhere, so the H1/H2 comparisons are null rather than ranked — the probe and agreement results carry the comparison, and the chapter does not promote them into claims about general agentic capability. No external replication and no independent review.

Preparing for memory

Cloning failed where it matters: the student reproduces training rollouts and fails new situations. One plausible reason is the memorisation hypothesis above: the policy reproduced training states without extracting the lookup–read–answer rule that would carry it to unseen combinations. The floor result is consistent with that reading but does not isolate it.

Chapter 27 asks whether that knowledge has to live in the weights at all. Give the agent an external memory — a store it can read at decision time — and the memorisation burden should, in principle, disappear; change the stored facts, and the question becomes what the agent does with stale knowledge and how a deletion propagates. The environment stays the same. The carrier of knowledge changes, and with it the trade-off the next chapter measures.

References

Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. and Cao, Y. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. International Conference on Learning Representations (ICLR). arXiv:2210.03629. https://arxiv.org/abs/2210.03629

Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N. and Scialom, T. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools. Advances in Neural Information Processing Systems 36 (NeurIPS). arXiv:2302.04761. https://arxiv.org/abs/2302.04761