← Distillation From First Principles

Synthetic Data Generation

Generating teaching examples at scale requires measuring what generation produces: correctness, duplicates, diversity, coverage, cost, and provenance. This chapter builds a bounded E3 generator with an audited quality comparison and no student training.

The generation study in this chapter produced 2,304 synthetic candidates in under half a second, and about 95% of them passed verification. Neither number says whether those candidates would teach anything. They were drawn from a space of 87,296 possible inputs; the generator spread its draws evenly across lengths, so the small short-length subspaces were sampled densely while the far larger long-length spaces were barely touched; and no student was trained on the result. Plenty is cheap; coverage, correctness, and provenance are what have to be earned.

Chapter 13 defined the teaching task. This chapter generates the teaching: examples at useful scale, with their correctness checked, their duplicates counted, their diversity measured, and their provenance recorded. The central question is:

How can synthetic examples be generated at useful scale without losing diversity, correctness, or legal clarity?

The answer this chapter defends is that synthetic data is not valuable because it is plentiful. Its value depends on the coverage, correctness, diversity, provenance, and rights of the generated examples — and on the information lost or distorted by the generation and rejection procedures. Quantity is the easiest property to measure and the least informative about teaching value.

Where synthetic teaching data comes from

Synthetic examples in this book come from three kinds of machinery. A program applies a known rule: the E3 generator draws lengths and symbols from a seeded RNG and computes targets with the deterministic oracle. A template turns parameters into prose: one of four instruction phrasings is filled with a seeded instruction, input, and format, and the target is still computed by the oracle. A teacher — real or stand-in — produces outputs from inputs: here, a deterministic scripted fixture, explicitly not a learned model, that emits valid outputs most of the time, repeats earlier outputs sometimes, and produces invalid outputs the rest of the time.

The distinction matters because each kind fails differently. Programs fail by narrowness: every example obeys the rule, so nothing outside the rule can appear. Templates fail by superficial variety: new phrasings over the same underlying distribution. Teacher stand-ins fail by imitating failure modes rather than exhibiting them: our scripted invalid outputs are hand-enumerated (wrong order, bad token, truncation), not the mistakes a real model would make. Naming the generator kind is therefore part of every candidate’s record, not metadata.

KD-020 and the seven kinds of synthetic data

Synthetic data (KD-020) is data generated by a model, program, or template rather than collected directly from its target environment. The registry distinguishes seven kinds, and they are not equivalent:

  1. Rule-generated examples (programs with oracles).
  2. Template-generated examples (parameterised prose or structures).
  3. Model-generated inputs (a model proposes the questions).
  4. Model-generated labels or answers (a model proposes the responses).
  5. Teacher-generated demonstrations (a teacher’s outputs as targets).
  6. Model-generated reasoning traces (Chapter 18’s subject).
  7. Recursively generated training data (a model trained on model outputs — the collapse regime discussed below).

A perfect rule-based oracle and a fallible learned teacher generate different kinds of evidence: the first certifies correctness by construction, the second exhibits the errors a student would inherit. Our study uses kinds 1 and 2 for correct examples and a scripted stand-in for kind 5 — and reports exactly that, because calling a scripted fixture “teacher-generated” without qualification would borrow credibility the experiment did not earn.

Synthetic does not imply: novel, correct, representative, independent, unrestricted, legally reusable, or useful for training. Each of those is a separate claim requiring separate evidence, and the audit in this chapter checks them one by one.

Programs, templates, and scripted teachers

The three generation modes share one contract and differ in one ingredient each:

  • A-programmatic draws a length uniformly from 4–8 and symbols uniformly from {3, 4, 5, 6} with a seeded RNG, then applies the oracle. Every candidate is correct by construction; the only risks are duplication (random re-draws collide) and narrow coverage.
  • B-template draws one of four templates (T-agree, T-polars, T-reverse, T-compact), a phrasing of the same reverse-complement instruction, and seeded inputs, then applies the oracle. Holding the task fixed while varying the phrasing isolates what templates contribute: presentation variety, not new behaviour. A run of 256 candidates uses each template roughly equally (seed 0: 60/70/54/72).
  • C-scripted-teacher draws a mode per candidate (seeded): valid with probability 0.7, duplicate of an earlier valid output with probability 0.15, invalid with probability 0.15 (cycling wrong-order, bad-token, truncated). It is deterministic, inspectable, and wrong on purpose in known ways — a fixture for testing the verification pipeline, not a stand-in for learned behaviour.

All modes share the E3 TaskSpec from Chapter 13 as their correctness contract, and every candidate carries its generator identity, seed, template (if any), bound variables, and output hash from birth.

The generation specification

A GenerationSpec states the full generation contract before a single example is produced: run identity, task identity and fingerprint, generator identity/type/version/hash, template identity and hash where applicable, configuration hash, seed, candidate budget, sampling description, verification procedure, duplicate policy, rights record and authorised action, runtime ceiling, and known limitations. It is a frozen dataclass, validated on construction, serialised with the same canonical JSON as the DataManifest records from Chapter 12 — deliberately the same provenance system, not a second one.

The task specification points at its dataset manifest by fingerprint; the generation specification points at the task the same way, so the chain task → data → candidate is one linked record, not three loose files. For this study’s frozen run (E3S-20261115T015728Z-576b8d), the programmatic arm’s spec records the E3 task fingerprint, the module hash of the generating code, the uniform-length sampling description, the 256-per-seed budget, and the RM-106 action; the template arm additionally records its template id and template hash per candidate. The run’s GenerationSpec records are stored in the manifest, so any candidate can be traced back to the exact rule, code, and configuration that produced it.

Freezing matters more than format. The study protocol (budget, seeds, templates, thresholds, quotas, comparison groups) was written before any candidate was generated, and the single controlled run that followed is the only run reported. A generator tuned until its preferred arm wins is not an audit; it is advertising with extra steps.

    flowchart LR
  S[GenerationSpec frozen] --> G[generate: program, template, scripted teacher]
  G --> V[verify: schema, oracle, duplicates]
  V --> A[accept with reason]
  V --> R[reject with reason]
  A --> L[provenance ledger keeps both]
  R --> L
  

Figure 1. Every candidate, accepted or rejected, ends in the ledger with its reason. A rejection audit that discards its rejects is advertising.

Sampling distributions

Let \(P_G(x,y)\) be the candidate-generation distribution: the probability that the generation procedure emits a given input-output pair. It is determined by templates, sampling weights, seeds, and generator behaviour — not by the task. Our programmatic generator draws lengths uniformly from {4,…,8}, so length 4 and length 8 are equally likely under \(P_G\); a generator that favoured short examples (say, length sampled proportional to \(2^{-L}\)) would put most of its mass on length 4 while the intended task spans all five lengths.

The generated data is then biased before any student training occurs, and no downstream metric can un-bias it: coverage lost at generation is lost, full stop. Concretely, our three seeds produced 2,304 candidates with per-seed acceptance 0.940–0.958 (2,192 accepted total); a short-favouring generator with the same budget would spend most of those draws where the evaluation needs them least — a fact only visible because the sampling distribution was written down first.

This is why the chapter insists on distinguishing \(P_G\) from \(P_{\mathrm{task}}\). The curated comparison set exists precisely to hold the alternative fixed: stratified by length (13/13/13/13/12 per seed), it represents the uniform-coverage choice against which any generator’s implicit distribution can be judged.

Rejection sampling and its cost

Let \(a(x,y)\in[0,1]\) be the probability of accepting a generated candidate. The accepted sample follows

\[ \frac{P_G(x,y)a(x,y)}{Z}, \qquad Z = \mathbb{E}_{(x,y)\sim P_G}[a(x,y)], \]

where the normaliser \(Z\) is exactly the acceptance rate. Rejection changes the effective distribution: accepted examples over-represent whatever the filter keeps. For independent proposals with a stable acceptance probability \(a\), the expected number of generations per accepted example is \(1/a\) — each halving of the acceptance rate doubles the generation bill. That relationship is exact for independent proposals and is reported as such.

Sequential deduplication is different and must not be conflated with it: whether the 200th candidate is kept depends on the previous 199, so there is no fixed \(a\) and no \(1/a\) law. The study reports both mechanisms separately — per-candidate rejection for correctness, sequential comparison for duplicates — because a single acceptance number would hide which one is doing the work.

Cost follows the same discipline. The reported unit is CPU time per accepted example: (generation time + verification time) / accepted count, measured with wall-clock timers on one thread. The study’s totals are small enough to state plainly: three seeds, 2,304 generated candidates, 2,192 accepted (rate 0.94–0.96 per seed), whole run under half a second — 0.006 to 0.060 ms per accepted example depending on seed. The scripted arm’s acceptance rate of 0.82–0.88 implies a generation multiplier of roughly 1.14–1.22×: about one extra generated candidate for every four and a half to seven accepted ones. No provider pricing exists here, so none is invented; a deployment that paid per token would meter differently, and the chapter says so instead of fabricating a price.

Diversity and duplication

Four measures, each with an explicit definition and a stated limit:

Unique-example ratio (unique inputs / candidates) is the first check. The scripted teacher’s duplication arm exists to make this number move: exact re-emissions are caught by identity comparison against every earlier candidate, not just neighbours.

Category coverage counts present bands: lengths {4,…,8}, template families, operations. Coverage is presence, not balance — the curated set’s quotas (13/13/13/13/12) are a balance rule on top of coverage, and the two are reported separately.

Symbol entropy (bits, base 2; zero-probability symbols contribute nothing) summarises the input-symbol distribution. Near 2.0 bits on a 4-symbol alphabet means near-uniform; it says nothing about order, structure, or meaning.

Near-duplicate rate uses token-set Jaccard similarity with a declared threshold of 0.9, counted over unordered pairs. On this tiny string space the threshold is deliberately loose — thousands of pairs qualify per pool — which is itself a finding: with four symbols and lengths 4–8, random draws are supposed to look alike. The count is a similarity signal, not a defect count, and the chapter refuses to treat it as one.

Lexical diversity is not semantic diversity. Two instructions can share no tokens and demand the same transformation; two identical inputs with different targets are a correctness problem, not a diversity success. The audit reports the measures it can compute and names the ones it cannot.

In this study the honest diversity story is thin, and the chapter says so: template use is roughly even (54–72 per template per seed), symbol entropy sits at ~2.0 bits everywhere (the generator is uniform by construction), and uniqueness runs 245–253 of 256 for the clean arms. Nothing here demonstrates rich coverage — the space has 87,296 possible inputs and the pools show 2,304 draws — which is precisely why Chapter 15 will need selection, not just generation.

The finite space of possible examples

The E3 input space is finite and exactly countable:

\[ N = \sum_{\ell=4}^{8}4^\ell = 256+1024+4096+16384+65536 = 87{,}296, \]

verified in code (the same 87,296 the Chapter 12 audit records). Finiteness has a concrete consequence: repeated sampling must eventually collide, even with a perfect generator. Under uniform independent sampling with replacement, the expected number of unique observations after \(n\) draws is \(N(1-(1-1/N)^n)\). That formula is an upper-bound reference, not a description of our length-uniform generator — stated as such — but it makes the structural point: generating more examples does not proportionally grow unique information, and at 768 draws per seed from 87,296 possibilities the pools sit deep in the regime where collisions are rare but inevitable.

Concretely, 768 uniform draws expect about 765 unique inputs and the full three-seed 2,304 draws expect about 2,274. Our generator is length-uniform rather than pair-uniform, which concentrates draws on the small length-4/5 subspaces: the same calculation applied per length band predicts about 249.6 uniques per 256-draw pool, against 245–253 observed. The gap between formula and observation is explained, not hand-waved — and both confirm the structural point: the observed duplicates are the birthday effect, not a generator defect.

Duplicates are the birthday effect, not a generator defect

Figure 2. Expected duplicate counts recomputed from the repository’s own function: rare at 768 draws, visible at 2,304, matching the pools. Arithmetic reference, not a measurement.

Basic quality verification

Every candidate passes through the same checks, in order: TaskSpec schema validation (ids in vocabulary, length bounds, EOS termination, no PAD), oracle comparison (output minus EOS against the deterministic transform), then duplicate detection. The duplicate count covers all generated candidates — including structurally rejected ones — while the accepted count covers only candidates satisfying the criteria, so an arm can report more unique inputs than accepted examples (e.g. seed 0 scripted-teacher: 225 unique inputs from 256 candidates, 222 accepted).

This denominator difference is definitional, not an error, and the manifest records both counts separately. Rejection reasons are recorded per candidate and never silently dropped: invalid-structure (bad token, missing or wrong termination, length violation) and oracle-mismatch (well-formed but wrong). Duplicates are counted at the audit level rather than as rejections — an exact re-emission that verifies correctly is accepted but flagged — with near-duplicate pairs reported separately at the stated Jaccard threshold.

  • invalid-structure (bad token, missing or wrong termination, length violation),
  • oracle-mismatch (well-formed but wrong).

Duplicates are not rejection reasons in this study: an exact re-emission that verifies correctly is accepted but counted (the scripted teacher’s duplicate arm exists precisely to exercise that path), and near-duplicate pairs are reported at the audit level. This is a design choice, not a theorem — a deployment pipeline might well reject duplicates outright (Chapter 15’s territory) — and the manifest records which choice was made.

Structural checks run before the oracle, so a malformed output is rejected as malformed rather than misdiagnosed as merely incorrect. A missing EOS is a structural failure — termination is part of the task contract — and a swapped-but-valid output is an oracle failure. The distinction matters because the fixes differ: one is a format bug, the other is a wrong answer.

Teacher-generated outputs and errors

The scripted teacher exists to produce the three things a correctness audit must handle and that clean generators never emit: valid outputs (accepted), duplicates (re-emitted earlier outputs, accepted but flagged), and invalid outputs in known flavours. Wrong-order swaps two adjacent symbols; bad-token substitutes an out-of-vocabulary id; truncation drops the tail including EOS. Where a corruption is a no-op (identical adjacent symbols), the output is oracle-correct and is labelled valid — correctness is a property of the output, not the process that produced it.

This is stated plainly because the alternative is misleading: nothing here was generated by a learned model, and no claim about model-generated error distributions follows from these fixtures. What the fixture does establish is that the verification pipeline distinguishes the three cases and records each one, which is exactly what a fixture is for. Teacher-generated supervision can in principle carry richer signals than labels alone — Hsieh et al. (2023) show that model-generated rationales paired with labels can train smaller models with less data — but that finding belongs to reasoning distillation (Chapter 18), is reported here only as motivation, and was not replicated: our scripted outputs carry no rationales at all.

The bounded E3 generation study

One frozen run, three seeds, 256 candidates per mode per seed (2,304 total), plus a curated 64 and a seeded-shuffle random 64 per seed drawn from training inputs only (never validation, never test). No student was trained; the comparison is quality, diversity, and cost — never downstream learning.

Per-seed results (accepted / unique / near-dup pairs / oracle-correct):

  • Seed 0 — A: 256/253, B: 256/248, C: 222/225; curated 64/64, random 64/64.
  • Seed 1 — A: 256/251, B: 256/245, C: 210/209; curated 64/64, random 64/64.
  • Seed 2 — A: 256/248, B: 256/249, C: 224/211; curated 64/64, random 64/64.

Acceptance rates: A and B 1.000 every seed (oracle-generated); C 0.867/0.820/0.875 (the invalid arm working as designed). Overall budgets: 768 generated per seed, 734/722/736 accepted (rates 0.956/0.940/0.958), at 0.006–0.060 ms per accepted candidate on CPU. Template use is roughly even (seed 0: 60/70/54/72 across the four templates). Curated sets hit their quotas exactly (13/13/13/13/12); the random baseline skews (seed 0: 9/9/15/17/14) — a small, honest demonstration that stratification guarantees what shuffling merely hopes for. Entropies sit at ~2.0 bits everywhere: the symbols are uniform, and uniformity is all entropy can see.

Generated versus curated

The comparison the study was built for: 768 generated candidates per seed against 64 curated and 64 random examples. On correctness the oracle-generated pools are perfect by construction and the curated and random sets are perfect by verification — the difference is not accuracy but guarantee structure: the pools earn correctness candidate-by-candidate (rejecting ~4–18%), while the curated set earns it by construction rule plus verification, with stratified length coverage the random set cannot promise.

On uniqueness the pools show 245–253 of 256 (A/B) and ~210–225 (C, where duplicates are intentional); both 64-sets are fully unique. On cost, generation plus verification runs to fractions of a millisecond per accepted example on CPU — the bill that matters here is the rejection overhead (1/acceptance rate), not compute.

What the study does not conclude is stated with equal care: no student was trained, so nothing here says which set would teach better. A smaller curated set with guaranteed coverage is cheaper to verify and easier to reason about; whether that translates into learning is a Chapter 15 question, requiring a separately authorised training comparison that this stage explicitly did not run.

Provenance of every candidate

Each of the 2,304 candidates carries: candidate and run ids, seed, mode, generation order, task id and fingerprint, generator identity, version and hash, template id and hash where applicable, bound variables (including the rendered prompt for template candidates), input, output, output hash, verification result and reason, duplicate flags, and the rights record. The full ledger ships as candidate-provenance.jsonl (2,304 lines); the manifest records the run-level aggregates, configuration hash, source hashes, and cost. Rejected candidates keep their records — a rejection audit that discards its rejects is advertising. Anyone with the repository can re-derive every number in the results tables from the ledger plus the frozen code; that is the test the provenance design was built to pass.

Model collapse and self-reinforcement

Shumailov et al. (2024) report that training on recursively generated content causes model collapse: tails of the original distribution disappear and defects compound across generations, shown in VAEs, Gaussian mixtures, and LLMs. The mechanism matters more than the slogan. Collapse is a property of recursive training — models trained on their own outputs, generation after generation, with feedback amplification and diversity loss at each step. It is not a property of generating synthetic data once, with verification, from a fixed task definition. Our study is firmly on the safe side of that line: one generation pass, oracle-verified, no student trained, no loop closed — and it must not be cited as evidence about collapse in either direction.

What our diagnostics can say about the risk is limited but real: the pools’ uniqueness ratios, near-duplicate counts, and entropy show whether a single generation step is already narrowing the distribution. Here they show mild narrowing (a handful of exact duplicates per 256 draws, heavy token-set overlap inherent to the tiny space) — worth monitoring across repeated rounds, not a finding of collapse. Recursive self-training is explicitly out of scope: no such experiment was run, and the chapter makes no claim about what would happen if one were.

Rights and generated content

The study ran under RM-106 (generate_and_audit_e3_synthetic, verified-permitted): self-created generators, seeded templates, a scripted teacher, no external models or data, CPU only, no redistribution. Synthetic does not mean unrestricted: generated outputs inherit the generator’s terms where a generator has them — a hosted model API typically restricts competing-model training on its outputs, which is why this study uses no such API at all. Personal or copyrighted content can in principle appear in model-generated text (not in our symbol strings, by construction), and training, redistributing, or competing are separate permissions, not inferences from generation rights. RM-104’s pending re-attestation is orthogonal to this study and remains open; nothing here trains a student on E3 data or expands any prior permission.

What synthetic data can and cannot establish

It can establish, as here: how much of a generated pool survives verification, how diverse it is under stated measures, what it costs per accepted example, and whether every candidate is traceable to its generator. It cannot establish: that the examples are representative of anything beyond the generator, that a student would learn from them, or that the same pipeline works at language-model scale. The curated-versus-generated comparison is a quality and cost comparison, not a teaching comparison — and presenting it as anything else would be exactly the volume-equals-value confusion this chapter exists to dispel.

Preparing for filtering and curriculum learning

Generation produced 2,304 candidates with known provenance and a measured acceptance rate near 95%. The remaining questions are selection questions: which candidates to keep, in what order to present them, and how to document the choice. That is filtering and curriculum — Chapter 15’s subject — and it starts from this chapter’s outputs: the candidate ledger with per-item reasons, the coverage matrices, and the curated baseline that filtering must beat to justify itself.

The staged sequence, in full, is: (1) define the desired behaviour; (2) define the input and output schemas; (3) define correctness; (4) define the evaluation metric; (5) define coverage and difficulty; (6) define leakage and rights constraints — all Chapter 13; (7) construct or generate examples — this chapter; (8) filter and order them — Chapter 15; (9) train and evaluate — later parts.

The published pipelines the chapter draws on map onto the same stages: Self-Instruct builds from 175 seed tasks, generates instructions, identifies classification tasks, generates instances input-first or output-first, then filters by ROUGE-L similarity (< 0.7), heuristics, and duplicates — 52,445 instructions and 82,439 instances in all, with an expert quality audit of 200 sampled instances finding 92% valid instructions, 79% appropriate inputs, 58% correct outputs, and 54% fully valid in every field (Table 2 of the paper). Unnatural Instructions expands 64,000 seed-elicited examples to ~240,000 by paraphrase. In both cases the task definition, the generation step, and the filtering step are separate operations with separate failure modes — which is why this book gives each its own chapter rather than one “data pipeline”.

References

Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D. and Hajishirzi, H. (2023). Self-Instruct: Aligning Language Models with Self-Generated Instructions. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13484–13508. https://aclanthology.org/2023.acl-long.754/

Honovich, O., Scialom, T., Levy, O. and Schick, T. (2023). Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14409–14428. https://aclanthology.org/2023.acl-long.806/

Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R. and Gal, Y. (2024). AI models collapse when trained on recursively generated data. Nature 631:755–759. https://doi.org/10.1038/s41586-024-07566-y

Hsieh, C.-Y., Li, C.-L., Yeh, C.-K., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.-J. and Pfister, T. (2023). Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Models. Findings of the ACL 2023. https://arxiv.org/abs/2305.02301