← Distillation From First Principles

Verifier-Guided Distillation

A verifier scores candidate reasoning traces and selects training data: outcome versus process checks, false positives, rejection sampling, best-of-n, and a controlled E3 comparison of unverified, outcome-verified, and process-verified supervision.

Two students in this chapter’s experiment were trained on traces selected by two different verifiers: one that checked only the final answer, and one that re-executed every step. Their results matched to three decimals on every seed. The verifiers had agreed on all 2,400 candidates, because every teacher trace with a correct answer here also had valid steps. A process verifier can only show its value where answers and reasoning come apart, and this chapter measures whether they did.

Chapter 18 built the apparatus for checking reasoning: an oracle that re-executes every step. This chapter puts that apparatus to work as a selector — a verifier that decides which candidate traces a student is allowed to train on:

How can a verifier improve the teaching signal without confusing selection with reinforcement learning?

The answer this chapter defends is careful and two-sided. A verifier can remove wrong supervision before training, and on this task outcome filtering is associated with a small gain (+0.025 over unfiltered on the mean, negative on one seed and inside the seed noise).

But process filtering adds nothing beyond outcome filtering — not because process supervision is useless in general, but because, on 2-step chains from this teacher, every correct answer already rides a valid trace. The verifier’s value is bounded above by the generator’s failure modes, and this chapter measures both sides of that bound. Where the generator fails rarely and uniformly, verification is cheap and nearly pointless; where it fails often or adversarially, verification is expensive and possibly decisive. Our task sits firmly in the first regime — reported, not regretted.

Why verification, and of what

Chapter 18 ended with an uncomfortable asymmetry: students learn steps teacher-forced (~0.90) but cannot generate them free-running (~0.06 valid). If the student cannot produce valid traces, perhaps it should not train on invalid ones. That is the verifier’s job description: score candidate (problem, trace, answer) triples and keep the ones worth learning from. The word “worth” does all the work, and the rest of the chapter is an attempt to define it without smuggling in the conclusion — because the easiest verifier to build (keep the traces with correct answers) decides every case identically to the hardest one on this pool (100% verdict agreement across 2,400 candidates — outcome-accepted and process-accepted sets coincide), and that fact, discovered rather than assumed, reshapes the experiment.

Eight related ideas must not be conflated along the way: outcome verification, process verification, learned reward modelling, deterministic executable checks, rejection sampling, best-of-n selection, supervised distillation on selected examples, and reinforcement-learning optimisation. The first two are checks; the third is a model of checks; the fourth is what this chapter implements; the fifth and sixth are selection rules; the seventh is what the students actually train on; the eighth is something else entirely, fenced off in its own section below.

KD-023 and the selection/optimisation boundary

Verifier-guided learning (KD-023) is the use of a verifier or reward model to score, filter, or rank candidate outputs. The registry places it among LLM-and-reasoning concepts, and the placement matters: a verifier scores candidates; what happens next determines the method. If selected candidates become supervised training examples, that is verifier-guided distillation — filtering, the subject of this chapter. If verifier scores become rewards that update a policy by gradient ascent on expected reward, that is reinforcement learning — a different method, owned by other chapters, explicitly not implemented here.

A numeric score is not a policy gradient. Selecting the top candidate is not PPO. The chapter repeats this because the literature it draws on (Math-Shepherd is used for PPO in its second application) makes the confusion easy, and because calling filtering “RL” would borrow rigour the experiment did not earn.

Outcome versus process supervision

For a candidate \(y = (r, a)\) (trace \(r\), answer \(a\)) given input \(x\), define an outcome verifier \(V_O(x, a)\) that checks the final answer and a process verifier \(V_P(x, r)\) that checks the intermediate procedure. The four cells:

Process Answer Meaning
valid correct fully verified — safe to teach
invalid correct THE dangerous cell: right answer, wrong reasoning
valid incorrect extraction or reporting failure downstream of sound reasoning
invalid incorrect unambiguously unusable

A complete check needs both: \(V_{P+A}\). Anyone who has read Chapter 18 recognises the second row: it is the fixture the chain_ok rule exists to catch — a correct answer with an invalid intermediate step, which an outcome check accepts and a process check rejects. The chapter’s central demonstration is exactly this cell: an imperfect verifier that mistakes outcome for process lets it through, and the confusion tables below count the cost.

None of this assumes a learned model. Our outcome verifier compares decoded digits against the oracle answer; our process verifier re-executes every step. Deterministic, inspectable, and — stated flatly — not a reward model. What transfers from the literature is the design: separate the two checks, measure each, and never let one impersonate the other.

Deterministic checks versus learned rewards

Three verifier kinds appear, and the chapter never lets them blur. deterministic_outcome_oracle and deterministic_process_oracle are total on this task: every well-formed trace gets a verdict, and the verdict is correct by construction against the oracle. Learned reward models (PRMs) are the alternative the literature uses at scale — Lightman et al. train step-level classifiers on 800k human labels; Math-Shepherd bootstraps them without humans — and the chapter deliberately does not build one.

The reason is scope honesty, stated upfront: a learned verifier would need its own training, calibration, and failure analysis, tripling the experiment while answering none of this chapter’s questions better. What a deterministic oracle cannot tell us is also stated: nothing about natural-language reasoning, nothing about generalisation beyond executable checks, and nothing about reward hacking against learned scores (our gaming demo is a hand-built fixture, labelled as such — evidence of a failure mode, not of learned exploitation).

The outcome verifier

Given a candidate row, extract the answer field (tokens between ANS and the final EOS; malformed or missing markers yield an explicit unverifiable verdict, never a silent default), decode the digits, and compare against the oracle answer recomputed from the prompt. The output record carries: verifier id and version, check type, parsed answer, expected answer, boolean verdict, a 0/1 score, the failure reason, and the candidate’s provenance. On the 12-fixture audit set the outcome verifier is correct wherever an answer exists to judge — and that sentence already contains its limitation: it judges answers, not reasoning.

The false-positive case is built in, not discovered: a valid trace with a corrupted final digit is rejected (correct), while the missing-step fixture — STEP marker dropped but answer digits intact — is accepted despite its expected-answer verdict being False (the single FP). Note the asymmetry the audit preserves: the invalid-step-correct-answer fixture, a correct answer reached through an invalid step, is a true positive for answer correctness. It would become a false positive only if the outcome verdict were used as a verdict on process validity — which is exactly the misuse the imperfect-verifier fixtures demonstrate below.

Checking every step

The process verifier replays the trace like an accountant: for each STEP block, parse the claimed input, operator, and result; recompute apply_op; compare. Then the chaining check: step 0 must start from the prompt’s problem, and each step must consume the previous step’s result — chain_ok. Then the answer check: the final field must match the last step’s result, closing the loop between procedure and outcome. A trace passes only if all three hold.

A process verifier that merely re-checks the final answer is an outcome verifier with a misleading nameplate; the fixture suite contains exactly that mislabelling (imperfect-outcome-as-process) and measures its cost: precision 0.250 against the process reference, three false accepts out of eleven should-rejects. The distance between 1.000 and 0.250 is the distance between checking and pretending to check. Skipped steps, duplicated steps, truncated tails, unparseable tokens, and wrong-length numbers each produce a distinct, recorded reason. The verifier also asserts a property it is easy to overlook: probe parameters are never modified (there are none — it is pure code, and the test suite pins that down by asserting the fixtures’ verdicts rather than the implementation’s internals).

Verifier quality, measured honestly

Twelve labelled fixtures, each with an independently decided expected verdict (decided when the fixture was written, not by running the verifier — the suite would be circular otherwise). Against the answer-correctness reference, the outcome verifier scores TP 2, FP 1, TN 7, FN 0 over 10 decided fixtures (plus 2 unverifiable: duplicated-step and copied-template carry no expected-answer verdict, so they enter neither numerator nor denominator): precision 0.667, recall 1.000, false-positive rate 0.125. The single false positive is the missing-step fixture — dropped STEP marker, intact answer, accepted against an expected False.

The correct-answer/invalid-process fixture is a true positive here: for answer correctness it is exactly what it claims to be. It would be a false positive only under the process reference — the cell the whole chapter warns about, demonstrated by the imperfect-outcome-as-process fixture below rather than by this table. Against the process-correctness reference, the process verifier scores TP 1, FP 0, TN 11, FN 0: precision, recall 1.000, FPR 0.000. On fixtures, the deterministic process check is perfect — which is exactly what “deterministic” promises and exactly what it limits: perfection against its own oracle, on cases designed to be checkable.

The imperfect verifiers earn their keep here. imperfect-outcome-as-process (mislabeled outcome checking) scores precision 0.250 and FPR 0.273 against the process reference: three false accepts out of eleven should-rejects. imperfect-skip-step-0 accepts any trace whose only flaw is at step 0 — including the chapter’s running corrupted example. These are fixture demonstrations of a failure mode (a plausible verifier that does not check what it claims), not measurements of a deployed system, and the chapter labels them as such in code (fixture: True), in tables, and in prose.

Rejection sampling changes the distribution

Let candidates come from \(P_G(y \mid x)\) and let the verifier assign scores \(v(x,y)\). For threshold \(\tau\), the acceptance rule \(a_\tau(x,y) = \mathbf{1}[v(x,y) \ge \tau]\) induces the selected distribution \(P_G a_\tau / Z_\tau\) — the same rejection-sampling mathematics as Chapters 14 and 15, now with the verifier as the filter. The consequences rhyme too: selection changes correctness, difficulty, length, coverage, and cost at once, and a more selective verifier is not automatically a better teacher. Under independent proposals with acceptance probability \(p\), the expected cost is \(1/p\) generations per accepted example — stated with the usual independence caveat, then replaced by measured counts wherever the experiment has them.

The numbers make the abstraction concrete: with the observed acceptance rate 0.967 overall (0.979/0.966/0.955 per seed), the expected cost is ~1.03 generations per accepted trace — verification is nearly free because there is almost nothing to reject. Had the teacher been weaker (say p = 0.1), the same arithmetic would demand 10 generations per accepted example plus 10 verification checks, and the chapter that studies that regime would look very different. Cost is a function of the generator’s failure rate, not of the verifier’s cleverness.

Verification cost is 1/p

Figure 1. Expected generations per accepted example: ~1.03 at the measured 0.967 yield; 2 and 10 at p=0.5 and p=0.1. Only p=0.1 is stated in the prose; p=0.5 is 1/p interpolation. Arithmetic illustrations, not measurements. Finite pools get exact accounting, not asymptotics: accepted, rejected, and duplicate counts, never a bare yield rate.

Best-of-n, with the bill attached

For each problem, generate \(n\) candidates, score all of them, keep the argmax with a predeclared tie-break (lowest candidate index — binary verifiers tie constantly, and the rule must be written down, not improvised). The audit ran 64 problems × 4 temperature-1.0 samples (256 generated, 256 verified) and selected 0 valid traces. Those 256 candidates came from a 50-update probe student sampling at temperature 1.0 on 64 train-split problems — a deliberately weak generator used only for the audit. The training pool came from the frozen 800-update teacher’s greedy decodes on 800 problems at ~96.7% validity. Different generator, different decoding, different n: the 0/64 and the ~96.7% describe different populations, so neither generalises to best-of-n in general.

The tie-break never even mattered: with a binary process verifier and zero valid candidates, every group ties at zero and the first candidate wins by default — a degenerate outcome the audit records rather than hides. Best-of-n multiplies generation cost by \(n\) and verification cost by \(n\); when the generator cannot produce anything worth selecting, that multiplication buys exactly nothing. The demo is audit-only (no student trains on it), and its lesson is economic: selection leverages generation quality; it does not create it. A stronger generator (or lower temperature) could change the arithmetic; the chapter reports the arithmetic it measured.

The frozen E3 comparison

Frozen run (E3V-20261115T214124Z-808d96). Option A from the protocol (teacher-generated candidates), because the audit supported it: greedy teacher traces verify at 0.979/0.966/0.955 per seed on 2-step problems (the 0.526/0.494/0.629 figures quoted in Chapter 18 describe that study’s teacher trace-validity diagnostic, a stricter full-trace measure on a different teacher — not this pool; against 0.000 on the abandoned 3-step pilot — the length cliff that killed the alternative of generating harder problems) — enough valid candidates to fill every arm without fabrication. Per seed, 800 candidates from 800 training problems (one each): outcome yields 783/773/764, process yields identical 783/773/764 (the collapse dissected below). Arms on the shared pool:

Arm Selection N/seed
A-unverified structurally valid only 800/800/799
B-outcome answer correct 783/773/764
C-process trace valid AND answer correct 783/773/764

Equal-size controls A’/B’ (seeded shuffle to |C|) ran as predeclared, since |C| ≥ 200 on all seeds. Students: the Chapter 18 width-32 architecture, EXACTLY 600 updates, batch 32, best-val restored after the full budget, identical inits across arms. Rights: RM-111 (verify_and_select_e3_reasoning_traces, verified-permitted), gate passed first. Held-out ID (2-step) + OOD (3-step) evaluation, greedy decoding, test ledger updated, no decisions followed from test outcomes.

Unverified versus outcome-verified training

A: 0.371 ID exact. B: 0.396. The +0.025 gap (per-seed: +0.021/−0.020/+0.073) is small, inconsistent in sign on seed 1, and about the size of the seed noise — reported as suggestive, not established. What is established is cheaper: filtering 2–5% wrong answers (2.1/3.4/4.5% per seed) lowered exact match on seed 1 and raised it on seeds 0 and 2, costs one deterministic check per candidate, and changes almost nothing else (the retained sets are 96–98% the size of the unfiltered pool — 97.9/96.6/95.6% per seed, 96.7% overall — and the coverage audit shows rejections spread across lengths rather than concentrated).

OOD: A 0.036, B 0.037 — indistinguishable, as expected: nothing in outcome filtering teaches longer chains. Token exposure is nearly identical across arms (~324k each), so volume confounds nothing here — the one ablation in this book where the token ledger is boring, and the chapter says so instead of manufacturing drama. The equal-size controls sharpen the point: A’ (0.383/0.358/0.308) tracks A within shuffle noise, while B’ (0.357/0.368/0.327) underperforms B on two seeds — subsampling a filtered set reshuffles its order, and order matters to optimisation.

Process-verified training

C: 0.396 ID, 0.037 OOD — identical to B to three decimals on every seed, because the selected counts coincide exactly (783/773/764) and every downstream metric matches bit-for-bit: every answer-correct teacher trace in this pool is also process-valid, and vice versa. Set identity is inferred from the shared-pool filtering code path plus identical counts and metrics — the frozen manifest stores per-arm aggregates but no arm-level content hashes, so the identity is strongly supported but not independently hash-verified. The audit that established this ran before selection, not after: outcome and process verdicts agree on 100% of the 2,400 candidates.

This is the chapter’s most important non-result. It does not show that process verification is useless in general. It shows that on 2-step chains from a memorising teacher, correctness implies validity — the generator’s failure mode is derailment (wrong steps, wrong answer together), never the dangerous cell (right answer, wrong reasoning). The dangerous cell exists in the fixtures and in the wild (Lightman et al. built PRM800K partly to hunt it), but this generator on this task never produces it, so a filter for it filters nothing.

The most important non-result: identical selections

Figure 2. Outcome-accepted and process-accepted sets coincide exactly (783/773/764 per seed). A verifier can only select among failures the generator actually commits. E3 verifier finding; set identity inferred from the shared-pool path plus identical counts and metrics. A verifier can only select among failures the generator actually commits. The equal-size controls confirm the package: A’ (0.383/0.358/0.308) tracks A, B’ (0.357/0.368/0.327) tracks B within shuffle noise — size is not the driver either. Selection, here, mostly reaffirms what generation already decided.

Does the student actually improve?

Modestly or not at all, depending on the comparison. Against unfiltered training: +0.025 ID, +0.001 OOD — a whisper, inconsistent across seeds, inside noise. Against the Ch18 answer-only baseline (0.093 on oracle answers): all verifier arms underperform — because they train on noisier teacher traces rather than oracle procedures, a confound the chapter names instead of burying: this experiment compares selection given a fixed flawed generator, not selection in the abstract. The forced-step diagnostic sharpens the picture: every arm scores ~0.95 teacher-forced (A 0.947, B/C 0.948) against ~0.36–0.39 free-running valid traces (A 0.363, B/C 0.388) — procedure knowledge is in the weights in every arm, unsurprising since even the unfiltered pool is 96.7% valid traces.

So the supervision did transfer procedure knowledge into the weights (forced step accuracy is real learning); it just does not survive free generation. Selection did not change what the model knows (forced-step accuracy 0.947 unfiltered, 0.948 filtered) and moved what it does only slightly (free-running valid traces 0.363 to 0.388). Forced and free are different verbs, and the chapter refuses to conjugate one as the other. Forced-step accuracy (~0.95 all arms) again exceeds free generation (~0.36–0.41 trace-valid), repeating Chapter 18’s gap inside every arm. The negative result stands as stated in the mission: a verifier can improve the retained dataset’s measured correctness (100% answer-correct in B/C vs 96.7% in A — 2320/2399 rows) while leaving student behaviour essentially unchanged — dataset statistics are not student outcomes.

Verification cost and coverage

Per candidate: one outcome check (parse + oracle compare) and one process check (re-execute ≤2 steps + chaining), both microseconds of CPU against ~milliseconds of greedy generation — verification is ~1% of candidate cost here, generation dominates 99-to-1. Selection yield: 96.7% retained overall (97.9/96.6/95.5% per seed), so rejection waste is negligible on this task because the teacher memorised the training set. The arithmetic that matters is the ratio: 2,400 verifications buy the removal of 17/27/35 verifier-rejected traces per seed (plus one structurally excluded row on seed 2 — 800 generated → 799 valid → 764 accepted — counted separately, not as a verifier rejection), and those removals move mean student exact match by +0.025 (per seed, −0.020 to +0.073).

Whether that trade is worth it depends on generation cost per candidate and on how much you trust a +0.025 whisper — the chapter supplies both numbers and declines to decide for you. Coverage: rejections (17/27/35 verifier rejections per seed, all “invalid process”, plus the one structural exclusion on seed 2) spread across lengths 23–29 roughly in proportion to the pool — no length band is emptied, no support lost. The honest accounting: this was a cheap filter on an easy pool. The expensive regime — low-yield generation where most candidates fail — is where verification cost bites, and this study does not reach it; the best-of-n audit (4× cost, 0 selections) is the closest glimpse, and it is labelled an audit, not an experiment.

Gaming the verifier, carefully labelled

Three strengths of claim, kept distinct. (1) A hand-constructed exploit: the skip-step-0 fixture accepts a trace whose only flaw the checker is told to ignore — demonstrates the possibility of verification gaps, proves nothing about any deployed system. (2) A generated exploit: not observed — the teacher never produced a correct-answer-invalid-process candidate in 2,400 draws, so there was nothing to study. That absence is itself measured (the fixture audit’s TN column counts what the generator never emits), not assumed.

(3) Learned reward hacking: not tested, not claimed; no RL loop exists in this book yet (Chapter 20 owns on-policy methods, and even there the plan is teacher scoring, not reward optimisation). The chapter’s position, stated plainly: our verifiers are too strong (deterministic oracles) for gaming to be interesting here and too narrow to conclude anything about gaming elsewhere. Both halves of that sentence matter.

Why this is not reinforcement learning

Two pipelines, different objects optimised. Verifier-guided distillation (this chapter): generate candidates → score → keep a subset → supervised cross-entropy on the kept set. The optimiser never sees a reward; it sees labelled tokens. The gradient is with respect to prediction error on fixed targets, not with respect to expected return under a stochastic policy — there is no baseline, no advantage estimate, no trust region, no sampled-trajectory variance to manage, because there is no policy optimisation happening at all.

Reinforcement learning would optimise the policy for expected reward, typically with sampled trajectories and a policy-gradient or value-based update — none of which occurs here. Math-Shepherd’s second application is RL (stepwise PPO with the process model as reward), and the chapter cites it as the contrast case that keeps our method honest: same verifier family, different optimisation, different claims. No policy gradients were computed in the making of this chapter.

Rights, provenance, and limits

Every candidate links its generator (frozen teacher recipe + seed), its verifier records (id, version, threshold rule), its selection verdict, and RM-111. Rejected candidates keep their records with reasons. Nothing is redistributed beyond the study artifacts; nothing personal exists in arithmetic traces. Limitations, without softening: one synthetic task, three seeds, tiny models, deterministic verifiers (not learned rewards), a memorising teacher whose failures are all of one kind, no demographic or subgroup dimension (none exists in the data), OOD as longer chains only, no replication, no independent review.

The strongest caveat is also the most interesting: because B and C selected equal-count sets with bit-identical downstream metrics (identity inferred from shared-pool filtering, not separately hash-verified), this study cannot distinguish outcome from process supervision as training interventions — only as verifiers (where the fixture audit does distinguish them: FPR 0.125 vs 0.000). That sentence is the chapter’s whole contribution in one line.

Preparing for on-policy distillation

Filtering fixed datasets is the end of the off-policy road: the remaining lever is the generator itself. Chapter 20 trains the student on its own trajectories with teacher scoring — on-policy data that narrows the forced-vs-free gap this chapter measured but could not close. It inherits the checker (as the scorer), the fixtures (as regression tests), the cost accounting (plus teacher-query costs), and the warning: selection optimises the dataset, not the student, and the two are not the same.

If this chapter’s lesson fits in one line, it is that a cleaner dataset is a cleaner input — what comes out depends on the training, and must be measured at the output. The natural next question — whether scoring the student’s own half-right trajectories teaches more than filtering someone else’s — belongs to Chapter 20, which starts from the ledger this chapter leaves behind.

References

Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, T., Lee, J., Leike, J., Schulman, J., Sutskever, I. and Cobbe, K. (2024). Let’s Verify Step by Step. ICLR 2024. https://arxiv.org/abs/2305.20050

Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C. and Schulman, J. (2021). Training Verifiers to Solve Math Word Problems. arXiv:2110.14168. https://arxiv.org/abs/2110.14168

Wang, P., Li, L., Shao, Z., Xu, R. X., Dai, D., Li, Y., Chen, D., Wu, Y. and Sui, Z. (2024). Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. arXiv:2312.08935. https://arxiv.org/abs/2312.08935