← Distillation From First Principles

Constructing Teaching Tasks

A task definition decides what the student should learn: inputs, outputs, correctness, metrics, leakage controls, and difficulty. This chapter builds TaskSpecs for E1 and the E3 precursor, plus a small instruction task, with prompt-leakage and ambiguity checks.

Take a constructed prediction that gets seven of a sequence’s eight tokens right. One metric scores it 0.875; another scores it 0.0: the single wrong token sat at the end-of-sequence position, so no valid output exists. Both numbers describe the same prediction. Which of them counts is not a training decision. It is settled earlier, when the teaching task is defined, and if it is settled wrongly no amount of data will repair it.

Chapter 12 established what the data is and whether it may be used. Chapter 13 asks what the data is supposed to teach: which inputs elicit the behaviour, what counts as a correct output, how difficulty is ordered, and how evaluation tells transfer apart from memorisation.

The central question is:

What exactly should the student learn to do, and how should the teaching examples represent that task?

The argument is that knowledge distillation succeeds or fails partly because of how the teaching task is defined. The task determines the inputs, the expected behaviour, the output format, the permissible variation, the difficulty, and the evaluation criteria. A perfectly trained model can optimise the wrong behaviour if the task specification is defective — and no amount of data volume fixes a task that measures the wrong thing.

The task comes before the examples

It helps to separate seven things that are easily conflated:

  • Task definition states what the student should learn.
  • Data provenance records where examples came from (Chapter 12).
  • Data generation produces the examples (Chapter 14).
  • Teacher supervision provides the training signal (Part II).
  • Training objective defines what is optimised.
  • Filtering selects which examples to use (Chapter 15).
  • Evaluation measures what was learned (Chapter 34).

Each has its own failure modes, and a failure at one stage cannot be repaired at another. A contaminated split is not fixed by a better loss; an ambiguous task is not fixed by more data; a proxy metric is not fixed by a larger teacher. This chapter owns the first item only: define the desired behaviour, the inputs and outputs that exhibit it, the rules for deciding correctness, and the metric that will measure it.

Defining what the student should learn

KD-044 — Teaching task is a specified input distribution, target behaviour, output format, and evaluation procedure used to elicit and assess knowledge transfer. Its essential properties are:

  • Intended behaviour: what capability the examples should elicit.
  • Valid inputs: what the student may be shown.
  • Valid outputs: what counts as a well-formed response.
  • Target semantics: what the output means, and how meaning is checked.
  • Difficulty: what makes some examples harder than others.
  • Coverage: which regions, cases, and edge cases must appear.
  • Evaluation metric: the function that scores behaviour.
  • Acceptance criteria: the bar for “the task is taught”.
  • Error categories: the known ways outputs fail.
  • Provenance and rights: which records authorise the task’s materials.

Note what is not on the list: the generator program, the teacher identity, the loss function, the dataset size. Those belong to generation, supervision, and filtering. A task specification answers a narrower question — what should the student learn? — precisely enough that data can later be generated, training can be run, and evaluation can be trusted.

KD-044 and the TaskSpec

The implementation (TaskSpec in examples/common/tasks.py, building on the DataManifest in examples/common/provenance.py) follows the book’s established spec pattern: a frozen dataclass, validated on construction, serialisable to plain JSON, with unknown fields rejected. It records identity and version (tasks change; the record must say which version is meant), a human-readable objective, input and target field schemas, the output format, the target-generation or labelling rule, the input distribution, validation rules, the evaluation metric, a difficulty definition, coverage requirements, the split policy, a link to the dataset manifest (by fingerprint, not by filename), the rights record and authorised action, and any known ambiguities and limitations.

The E1 task specification, built by e1_task_spec from recorded fingerprints, reads in full:

Field E1 value
Task ID / version e1-four-class-spiral / 1.0
Input schema point: 2-D float vector, finite coordinates
Target schema label: integer in {0, 1, 2, 3}
Output format label
Target rule Generating arm index; teacher labels are supervision, not truth
Metric Accuracy; cross-entropy and ECE diagnostic
Split policy Per-seed: train=seed, val=seed+1000, test=seed+2000; 1200/400/1000
Manifest link Dataset fingerprint + config hash (content-bound, not path-bound)
Rights RM-101, train_and_evaluate_e1

Three conventions keep it honest. First, the target rule is prose, not code: “reverse the input and apply the complement map” names the behaviour so a reader can judge it; the oracle implementation is referenced, not embedded. Second, the manifest link is a fingerprint comparison, not a path: the task is bound to content, so regenerating identical bytes under a new filename does not break the link and silently swapping bytes cannot preserve it. Third, the rights reference is checked by the same gate that guards training (check_action), so a task that names a prohibited or unknown action fails validation before any example is generated.

Inputs, targets and output contracts

A task’s output contract states the shape a response must take, and it is enforced structurally — before any question of correctness arises. The implementation distinguishes four formats: class labels, token sequences, JSON objects, and free text. Each carries its own validation:

Format Structural check
Class label Integer within the declared label set
Token sequence Integer ids, vocabulary membership, length bounds, EOS termination, no PAD
JSON object Parses; required keys present, no extra keys; values structurally valid
Free text Non-empty (and only this format tolerates extra prose)

The strictness is deliberate and task-specific. The E3 sequence task rejects trailing text after EOS because the decoder’s stopping behaviour is part of what is taught; a free-text instruction task permits it because verbosity is not the measured behaviour. In both cases the contract states the rule in advance — allow_extra_text: false is a field in the spec, not a judgment call at evaluation time. A valid output format is not proof that the answer is correct; it is proof that the answer is gradable, which is the prior question.

The formats compared side by side:

Format Structural check Semantic check Ambiguity risk
Class IDs Integer in declared set Matches ground-truth label Low; fixed vocabulary
Class names String in declared set Matches ground-truth label Medium; synonyms, casing
Token sequences Vocab, lengths, EOS, no PAD Oracle transform equality Low for canonical tasks
Structured JSON Parses; required keys, no extras Field values satisfy oracle Medium; key drift, types
Free text Non-empty Human or model judgment High; paraphrase, verbosity
Code Parses and runs Passes functional tests High; equivalent programs
Tool calls Schema-conformant invocation Achieves the intended effect High; side effects

Each row separates the two questions the chapter insists on: can the output be graded at all, and does it exhibit the required behaviour? Later chapters inherit this table: structured generation (Chapter 25) lives in the JSON row, tool-use trajectories (Chapter 26) in the last.

Correctness and evaluation

Five measures are easily confused, and the chapter keeps them separate because they disagree in practice:

  • Structural validity. Does the output parse and conform to the schema?
  • Semantic correctness. Does it mean the right thing, regardless of form?
  • Exact match. Is it byte-identical to a canonical reference?
  • Teacher agreement. Does it match the teacher’s answer?
  • Task success. Does it satisfy the independent requirements?

A JSON response can be structurally valid and semantically wrong (right keys, wrong transformation). A semantically correct response can fail exact match (an equivalent answer in different words — which is why exact match suits canonical symbolic tasks like E3 and would not suit open-ended generation). A teacher’s answer can be copied perfectly and still be wrong, as every chapter since Chapter 05 has warned: agreement is a diagnostic, never the objective.

The worked demonstration in this chapter makes the distinction unavoidable. A prediction that gets 7 of 8 E3 tokens right scores token accuracy 0.875 yet exact match 0.0 — the single wrong token is the EOS position, so the whole sequence fails. Token accuracy measures per-position imitation; exact match measures task completion. The metric must match the intended behaviour, or training optimises the proxy and evaluation reports the proxy’s success as the task’s.

Token accuracy 0.875, exact match 0.0

Figure 1. The demonstration drawn as positions: seven correct tokens average 0.875, but the wrong final token means no valid output exists. Worked demonstration, not an experimental result.

Task distributions

Let \(P_{\mathrm{task}}(X,Y)\) be the intended distribution of inputs and target behaviour. Two derived distributions matter for teaching:

  • \(P_{\mathrm{teach}}(X,Y)\): the distribution the teaching examples are drawn from — possibly biased toward common, easy, short, or format-clean cases.
  • \(P_{\mathrm{eval}}(X,Y)\): the distribution the evaluation draws from.

As Chapter 12 showed, a finite teaching set approximates the task distribution, and its quirks become part of what the student learns.

Covariate shift, label shift, and concept change are defined in Chapter 12 with the same vocabulary this chapter uses — including the worked label-shift toy calculation (90% on class A, 60% on class B: 75% balanced versus 63% at 90% class-B deployment, drawn as bars in that chapter’s Figure 2) and E1’s uniform-balance consequence. The definitions are not repeated here because the task-construction question is a different one: given that those mismatches exist, what does the teaching distribution declare?

A teaching set can be heavily biased without being wrong: oversampling short sequences early in a curriculum, for example, is documented practice rather than contamination — provided it is declared as a teaching choice (Chapter 15’s subject), not smuggled into the evaluation. Formally, teaching draws from \(P_{\mathrm{teach}}\) while claims are about \(P_{\mathrm{task}}\); the two coincide only by construction or by luck. Declaring the teaching distribution — in the spec’s input distribution and coverage requirements — is what makes the difference auditable.

The failure mode is silent mismatch: teaching on \(P_{\mathrm{teach}}\) while claiming the results describe \(P_{\mathrm{task}}\). The coverage matrix in this chapter exists to make the teaching distribution visible: class counts, length bands, symbol frequencies, format variants, and edge cases, each counted from the actual splits. What the student is likely to learn is a function of what it was shown; the matrix is the honest statement of what that was. For E1 the matrix is small enough to show in full — classes × splits with exact audited counts:

Split Class 0 Class 1 Class 2 Class 3 Total
Train (per seed) 300 300 300 300 1200
Validation (per seed) 100 100 100 100 400
Test (per seed) 250 250 250 250 1000

For E3 the matrix is length band × split, with symbol frequencies alongside (seed 0: 4162/4173/4115/4200 over 16,650 input symbols). In both cases the matrix is computed from the splits, not asserted from the generator: the audit counts what exists rather than trusting what was intended.

E1 classification task design

The E1 task, reconstructed from the recorded generator and the Chapter 12 audit (no retraining, no regeneration for training):

Aspect Specification
Input 2-D float point, finite coordinates
Output Label in {0, 1, 2, 3} (spiral arm index)
Target rule Ground-truth arm from the generating seed; teacher labels are supervision, not truth
Distribution Four interleaved spiral arms, radius 0.15+0.85u, Gaussian noise 0.08; uniform class balance
Metric Accuracy (primary); cross-entropy and ECE diagnostic
Splits Per-seed: train=seed, val=seed+1000, test=seed+2000; 1200/400/1000
Difficulty Complexity proxy: radius bins (inner turns denser); seed 2 hardest empirically — the proxy describes, it does not explain
Coverage All four classes, all splits; zero recorded overlap
Rights RM-101, train_and_evaluate_e1

The critical distinction is in the target row: a model predicting the teacher’s label is not the same as a model satisfying the ground-truth task. On this near-perfect-teacher task the two coincide numerically, which is precisely why the distinction must be stated rather than discovered — elsewhere they will diverge, and only the specification says which one counts.

E3 structured sequence task design

The E3 precursor task (protocol 1.1, audited in Chapter 12):

Aspect Specification
Input alphabet Symbol ids {3, 4, 5, 6}, lengths 4–8
Output alphabet Same symbols plus EOS=2; PAD=0 never predicted; BOS=1 conditions, never predicted
Target rule Deterministic oracle: reverse input, map 3↔6, 4↔5
Metric Oracle exact match (primary); token accuracy and NLL diagnostic only
Length handling Exact match reported by length band (compounding error is length-dependent)
Splits Per-seed disjoint by rejection; 1200/400/1000
Error categories Syntax errors, invalid tokens, wrong ordering, wrong substitution, wrong termination
Rights RM-104 (protocol-1.1 re-attestation pending; carried, not resolved)

Exact match is appropriate here because the target is canonical: one input admits exactly one correct output. The chapter states the converse explicitly — exact match would not suffice for open-ended generation, where paraphrase is correctness — so that the metric is not cargo-culted into Chapter 17’s territory. Error categories matter because they turn failures into diagnoses: a wrong-substitution error and a wrong-termination error imply different student deficiencies and different teaching responses.

Audit grounding

Chapter 12 performed the read-only retrospective audits these specifications are bound to: regeneration reproducing every recorded fingerprint byte-identically, zero duplicates within or across splits, E1 class balance exact, the E3 length mix and near-uniform symbols, filter acceptance mean 0.9975, and rights RM-101/RM-104 for their recorded actions. The method — regeneration, fingerprint comparison, duplicate and coverage tabulation, rights check — belongs to that chapter. What belongs here is what the task specification binds to: the per-seed fingerprints the E1 spec resolves, and the exact limits of what the E3 record can quote.

Seed Regeneration matches Within-split dups Cross-split overlap Train balance
0 yes (360a97f2…) 0 / 0 / 0 0 / 0 / 0 [300, 300, 300, 300]
1 yes (6e7041ef…) 0 / 0 / 0 0 / 0 / 0 [300, 300, 300, 300]
2 yes (0b65d8fe…) 0 / 0 / 0 0 / 0 / 0 [300, 300, 300, 300]

Validation is [100, 100, 100, 100] and test [250, 250, 250, 250] every seed. The audit confirms what the task design assumes: balanced classes, disjoint splits, reproducible bytes — and nothing more. It does not prove the historical runs executed this code path, only that the recorded parameters reproduce the recorded data.

For E3: 1,200 generated per seed × 3 seeds = 3,600 total. The length mix (all splits) is seed 0: {4:237, 5:472, 6:563, 7:660, 8:668}, with similar mixes for seeds 1–2; input symbols are near-uniform (seed 0: 4162/4173/4115/4200 over 16,650 symbols). The manifest field generation.num_targets = 1200 is the per-seed count from the final seed loop, not the three-seed total — recorded here so the field cannot mislead. Only the aggregate target hash was kept, so the per-example chain below the aggregate is illustrative, a limitation the specification records rather than papering over. The condensed E1 datasheet lives in experiments/E1-classical/audit-12/ and the full E3 provenance chain in experiments/E3-sequence-precursor/audit-12/; this chapter binds to both by fingerprint instead of restating them.

Instruction templates and leakage

An instruction template turns parameters into prose: “Transform the following {input} according to {instruction}; respond in {format}”. The template is code that handles untrusted content, and it must be audited like code. The failure mode is prompt leakage: the rendered prompt exposes the target, the answer key, evaluation metadata, filenames, or split-identifying markers, so the student scores by reading the question paper rather than learning the task.

The implementation (PromptTemplate) renders with explicit variables and checks every rendering against a secrets list — target values, answer keys, filenames, split markers. Anything undeclared that appears in the output is leakage, and the check fails closed: worked examples whose outputs are intentionally shown must be declared as such, not smuggled past the audit.

A deliberately leaking template (Answer: {target} embedded in the prompt; evaluation metadata as a source field naming a held-out file) is refused by the check in the test suite, while a clean template rendering the same task passes. The rule is data-flow, not keyword matching: it is the values that must not appear, because renaming “answer” to “response” fools a keyword search and fools no one else.

A concrete small task makes the contract tangible. Instruction: reverse_complement. Input symbols: [3, 4] (A, B). Deterministic oracle: reverse to [4, 3], complement 4→5 and 3→6. Output JSON:

{"operation": "reverse_complement", "input_sequence": [3, 4], "transformed": [5, 6], "valid": true}

The schema requires exactly the keys operation, transformed, valid (plus the echoed input); anything else is rejected as malformed before correctness is even considered. This is the book’s smallest complete instruction task: natural-language instruction in, schema-validated transformation out, oracle-decidable correctness — and it fits in a unit test rather than a training run.

The same separation — task definition, then generation, then filtering — structures the largest real-world instruction pipeline the chapter draws on. Self-Instruct (Wang et al., ACL 2023) starts from 175 seed tasks, generates instructions, identifies classification tasks, generates instances input-first or output-first depending on type, and filters by ROUGE-L similarity (< 0.7 against existing instructions), heuristic validity checks, and exact/near-duplicate removal — 52,445 instructions and 82,439 instances surviving.

Its quality audit is sobering: 92% of instructions valid, but only 58% of instance outputs fully correct. Definition, generation, and filtering are three different operations with three different failure modes, and the pipeline’s value comes from keeping them separate: Chapter 13 owns the definition, Chapter 14 the generation, Chapter 15 the filtering and ordering. Nothing here trains on Self-Instruct data or reproduces its results; the chapter uses its published pipeline and measurements as the motivating precedent.

Ambiguity review

Some examples should not be trained on, and the task specification must say what happens to them instead. The ambiguity review takes cases with at least two readings — two labels for one input, contradictory instructions, missing required fields, an output format admitting incompatible interpretations, an incomplete transformation rule — and records a disposition for each: accept, clarify, relabel, reject, or human review, each with a written rationale. A missing rationale fails validation, because an unexplained disposition is not a review.

Two principles govern. First, a deterministic oracle cannot resolve ambiguity that lives in the task definition: if the specification does not determine the answer, no labelling rule creates determinism, it only hides the choice. Second, ambiguous cases are quarantined from the historical datasets — the fixtures live in the task-validation suite, not in E1 or E3, and nothing in the recorded experiments is relabelled after the fact. The review’s job is to protect evaluation from meaningless agreement: two annotators (or teacher and student) agreeing on an ambiguous example proves nothing, and the chapter refuses to let it.

Difficulty calibration

A teaching task should state what makes examples hard, with the humility that complexity measures are not empirical difficulty. For E1, the documented proxy is radius: inner spiral turns pack arms closer together, so a radius-binned histogram (edges at 0.5 and 1.0, counts reported per split) describes the complexity mix the student sees. For E3: output length (error compounds per position), maximum repeated-symbol run, and symbol entropy per sequence.

Each rubric is demonstrated on training or validation partitions only — never tuned against held-out test outcomes — and each is presented as a description of the material, not an explanation of model behaviour. Seed 2’s hardness across E1 arms, for instance, is a measured fact the radius proxy does not explain, and the chapter says so rather than fitting the rubric until it does.

The E1 radius rubric in tabular form (edges at 0.5 and 1.0; bin counts are computed per split by difficulty_bins):

Bin Radius range Meaning
0 [0, 0.5) Inner turns; arms closest together
1 [0.5, 1.0) Middle turns
2 [1.0, ∞) Outer turns; arms most separated

Coverage and edge cases

The coverage matrix for E1 is inherited from the audit: uniform class counts per split, radius-bin counts, zero overlap. For E3: length bands, symbol frequencies, and the recorded 2,600-per-seed example space against 87,296 possible strings. The task adds what audits cannot: required edge cases with prescribed behaviour.

Empty inputs where prohibited must be rejected, not guessed; minimum (4) and maximum (8) lengths must be valid; repeated symbols, invalid tokens, unknown operations, missing fields, extra fields, and truncated output each map to respond correctly, return a defined error, reject, or ask for clarification where the task permits it. Edge cases are robustness tests, not necessarily training data: requiring graceful refusal does not mean teaching refusal by example, and the chapter keeps the two separate so that test coverage is not mistaken for a training recipe.

The metric consequences, summarised:

Metric Measures Fails to detect
Structural validity Gradability (parses, schema) Wrong content in valid form
Token accuracy Per-position imitation Whole-sequence failure (0.875 vs 0.0 above)
Exact match Canonical task completion Paraphrase-level correctness
Teacher agreement Imitation of the teacher Teacher error copied perfectly
Oracle task success Independent requirements Nothing about the task; the cost is needing the oracle

Optimising the wrong metric

The demonstration is small, deterministic, and labelled as such — not an experimental result. Reference: [5, 3, 3, 3, 3, 6, 5, 2]. Prediction: [5, 3, 3, 3, 3, 6, 5, 3] — identical except the final token, 3 instead of EOS. Token accuracy: 7/8 = 0.875. Exact match: 0. The prediction imitates nearly the whole teacher trajectory and completes nothing: the wrong final token means no valid output exists.

A training loop rewarded for token accuracy would keep this behaviour and an evaluation reporting token accuracy would praise it, while the task — produce the transformed sequence, terminated — fails on every such example. The moral is directional: choose the metric from the task definition outward, never from implementation convenience inward, because the student will optimise whatever you measure and the metric will call it success.

Rights and provenance

Every task component has a rights status: prompts, examples, reference outputs, generated labels, templates, teacher demonstrations. The TaskSpec records the rights record and authorised action for the task as a whole, and the read-only validation path checks them through the production gate before anything else happens — including the deliberately failing fixture EXAMPLE-UNVERIFIED-001 (an illustrative external source with unknown licence requesting student training), which is refused exactly as an unknown source must be.

Permission to train on a dataset is not permission to transform it, redistribute it, or build a competing model with it; each is a separate question for the rights record, not an inference from access. The E1 specification points at RM-101, the E3 specification at RM-104 with the pending 1.1 re-attestation carried forward unchanged: the task description does not expand, resolve, or backdate either record, and the detailed legal analysis remains where it belongs, in Chapters 36–39.

Validating a teaching task

The read-only validation path ties the pieces together without training anything: load a TaskSpec, resolve its DataManifest by fingerprint, check input and target schemas, enforce output-format rules, confirm split counts and disjointness against the audit, run the leakage checks, demonstrate ambiguity dispositions, compute coverage and difficulty summaries, wire the metric names to their implementations, and verify rights references through the gate. Where the historical record cannot satisfy a newly introduced check — E3’s per-example teacher provenance, for instance — the validator reports the gap instead of inventing the missing evidence. That asymmetry is the whole point of a read-only audit: the present can be strict only by being honest about the past.

Limits of this chapter’s evidence

This chapter trains nothing. Its task specifications are reconstructed after the fact from recorded generators and Chapter 12’s audits; they show that the records describe the data, not that the historical runs executed these specifications. The token-accuracy example, the leaking template, the ambiguity cases, and the unverified-source fixture are illustrations built for the validation suite. The difficulty rubrics describe the material and do not explain model behaviour; seed 2’s hardness in E1 remains unexplained. The Self-Instruct figures are its authors’ own audit. E3’s per-example teacher provenance was never recorded, and the RM-104 re-attestation for protocol 1.1 remains pending.

Preparing for synthetic data generation

Task definition ends where generation begins. The staged sequence is:

  1. Define the desired behaviour (this chapter).
  2. Define the input and output schemas (this chapter).
  3. Define correctness (this chapter).
  4. Define the evaluation metric (this chapter).
  5. Define coverage and difficulty (this chapter).
  6. Define leakage and rights constraints (this chapter).
  7. Construct or generate examples (Chapter 14).
  8. Filter and order them (Chapter 15).
  9. Train and evaluate (later parts).

The specification now states what to teach, how correctness is judged, which inputs are covered, and what is permitted — but it contains no generated examples beyond fixtures, no teacher outputs, and no curriculum ordering. Chapter 14 takes the specification as input and produces teaching data at scale: sampling, diversity, deduplication, quality control, and cost. Chapter 15 then orders it. The pipeline is deliberately staged so that each chapter’s failures are diagnosable at the stage that caused them: bad teaching is a task defect, bad examples are a generation defect, and bad order is a curriculum defect. Conflating them is how distillation projects end up tuning the student for a problem the task never posed.

References

Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D. and Hajishirzi, H. (2023). Self-Instruct: Aligning Language Models with Self-Generated Instructions. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13484–13508. https://aclanthology.org/2023.acl-long.754/