Part IV — Debugging Models

The test that passes on Tuesdays

Chapters 19–20 certified the input: bytes complete, ledger balanced, nothing cut. And the split-shipment fixture still flickers — 9/12 Monday, 5/12 Wednesday, same bundle, same revision. The engineer re-runs the failing case once, watches it pass, and closes the ticket. Friday it reopens. Nothing changed except the draw.

Concrete failure. At temperature 0.7 the refund answer cites 4.2 in six of ten runs and the general policy in four — same bytes, same weights, four different wrong amounts shipped to four different customers. The single-run “repro” measured a sample, not the system. The debugging object was never one output; it is the distribution over outputs that this input plus these parameters define.

OBSERVATION: frozen bundle at temperature 0.7 yields mixed citations across 20 trials (MEASUREMENT, seed swept, revision pinned); at temperature 0 the citation stabilizes but the amount still varies 2/20. HYPOTHESIS H1 (sampling spread): sampling entropy explains the flicker — deterministic settings collapse it. HYPOTHESIS H2 (bimodal competence): the distribution is genuinely two-peaked — the system holds both answers and picks by draw, so no sampling fix rescues it (repair is input, selection/verification, or weights). HYPOTHESIS H3 (serving plumbing): the flicker follows runtime numerics or execution configuration — batch shape, sequence slicing, tensor-parallel ordering, speculative decoding, scheduling — not the sampling parameters as coded. Same nominal params, different plumbing. HYPOTHESIS H4 (version drift): the flicker follows a silent model or runtime revision change — the “same” name on different weights or infrastructure. INFERENCE: none yet — only a seed-swept trial series with pre-written distributional FORECASTs separates spread from bimodality from plumbing from drift.

This chapter’s question: when the same program yields different answers, how do you debug the distribution instead of chasing the draw?

Why “it passed on retry” fails first

The obvious move — re-running once and accepting a pass — fails because a single draw from a mixed distribution carries almost no information about the distribution. Four sampling traps:

1. Single-run reliability. “It works” from n=1 at temperature > 0. The next customer draws again. Reliability is a rate with spread, never a sighting.
2. Temperature folklore. “0 means deterministic” cited as law. He (Thinking Machines Lab, lab report — not peer-reviewed) sampled 1,000 completions at temperature 0 of Qwen3-235B-A22B on a single “Tell me about Richard Feynman” prompt and got 80 distinct outputs as text identity (a semantic-cluster count would be lower — the 992× “Queens, New York” vs 8× “New York City” split at token 103 is near-paraphrase), first diverging at token 103 — greedy decoding, still not deterministic (He, Thinking Machines Lab, 2025). The demonstrated cause is that inference kernels are not batch-invariant: reduction orders depend on the forward pass’s batch size and sequence slicing, which depend on how much other traffic the server is handling at that instant. The mechanism has since been corroborated by serving-framework adoption (SGLang and vLLM ship batch-invariant paths; Microsoft’s LLM-42 preprint confirms the diagnosis while contesting the prescription — see the research lineage below), but the 80/1,000 rate is unreplicated: one setup, unstated GPU/driver defaults, no independent rerun found. Treat it as a bounded existence proof for one mechanism, not a general empirical rate. Actual determinism depends on the serving stack and changes over time — attribute such behavior to the reader’s own measured runs on their own endpoint, never to this book. Measure; do not quote.
3. pass@k misreading. “pass@10 is 95%, so we’re fine” — while deployment ships one draw (pass@1). pass@k measures whether any of k draws succeeds; it upper-bounds what reranking or selection could capture, and it never describes the shipped behavior. Reading it as reliability is score-as-diagnosis.
4. Seed theater. Setting one seed once and declaring reproduction. One seed is one draw with paperwork. Reproduction of a distribution needs a seed sweep with recorded spread.
5. Mean-without-shape reporting. “Accuracy 70%” over a bimodal split (half perfect, half catastrophic) reported as uniform mediocrity. The mean hides the two attractors; the histogram shows where repair actually applies.

OPINION: sampling parameters are code — temperature, top-p, max-tokens, seed handling ship to production exactly like prompt text. Unpinned, unlogged, or “left at default,” they are unreviewed code running the revenue path. Review them like code: pinned, diffed, gated.

Chapter 17 set the stance — with an opaque, stochastic system, repeated behavior is the evidence, because the interior cannot be inspected and one draw cannot be trusted. This chapter makes that stance operational: how to collect the draws, read their shape, and compare distributions across interventions.

The mental model: Distributions as the debugging object (book object #3, top.txt). The unit is not the answer but the answer-distribution: shape (stable vs. bimodal vs. long-tail wrong), rate (pass@1 with counts over N trials), and movement under parameter changes. Interventions predict distributional movement — narrower, relocated, unchanged — and single draws never confirm or kill anything.

Three sources of variation, one diagnostic question

Observed variation has three distinct provenances, and the debugger’s first task is determining which is active — not merely measuring that variation exists:

1. SAMPLING STOCHASTICITY (H1). The draw itself: temperature, top-p/top-k/min-p truncation, seed handling. Intended behavior of the generator; pinned params plus swept seeds separate it. Seeds govern the sampling RNG, never kernel numerics.
2. NUMERICAL / RUNTIME NONDETERMINISM (H3). Batch-shape-dependent kernel dispatch, sequence-slicing and KV-cache boundaries, tensor-parallel reduction ordering, speculative-decoding verification drift, scheduling and co-tenant load. The program is deterministic; the serving composition is not — from the user’s viewpoint by construction.
3. HIDDEN SYSTEM / VERSION VARIATION (H4). Silent weight, infrastructure, or configuration drift under a fixed name. Dominates all of the above across deploys; needs its own paired-revision probe because it mimics H3 exactly.

This split is old wine the field already bottled: flaky-test taxonomies classify this flicker as order-dependence plus randomness, and statistical debugging has always sampled many cheap runs and ranked predictors (Luo et al.; Liblit et al.). What is genuinely new here: high-dimensional outputs with no canonical equality, semantic equivalence as a measurement step, generative diversity as intended behavior to separate from unintended variation, serving variables the debugger cannot observe, and silent version drift.

Behavioral signature: the defined projection you actually compare

Byte equality is one possible signature — the right bar only inside a pinned model×runtime×hardware envelope with deterministic controls on. Everywhere else, the honest bar is a behavioral signature: the task-relevant observable properties used to decide whether two outputs are equivalent for the debugging question being asked. Fix it before trials, or the histogram is a Rorschach test:

BYTE EQUALITY (strongest, envelope-local)
  → token-sequence equality → semantic-cluster equality → task-success equality (weakest, most transferable)

Candidate components: exact text, schema validity, task success, claim set, citation support, tool selected, action chosen, semantic outcome class, invariant satisfaction. Two outputs may differ in wording while citing the same evidence, satisfying the schema, and passing the same invariant — same signature, no diagnostic difference. Two similar-looking outputs may belong to different classes. Semantic clustering (bidirectional-entailment grouping before counting, after Farquhar et al.) is the principled version of the chapter’s fixed classifier; it needs its own judge plus its own error rate, so prescribe it at the claim-set level, not for every series.

from dataclasses import dataclass

@dataclass(frozen=True)
class Signature:
    schema_valid: bool
    supported_claims: int
    unsupported_claims: int
    action: str | None
    task_passed: bool

def behavioral_signature(result) -> Signature:
    return Signature(
        schema_valid=result.schema_valid,
        supported_claims=result.supported_claims,
        unsupported_claims=result.unsupported_claims,
        action=result.action,
        task_passed=result.task_passed,
    )
# Group the N trials by Signature, not by raw text:
# the representation chosen for the outcome determines
# what distribution you think you are measuring.

The signature is a Design-by-Contract postcondition at the chosen strength (Ch8) — and it is what Ch23 compares across revisions and Ch42 aligns across trajectories. Define it once here; they inherit it.

Deterministic-controls hierarchy: what is pinned before N means anything

Run this ladder top-down before interpreting any series. No API exposes every rung; the debugging question at each is which sources remain UNKNOWN (Unknown ≠ Zero):

CONTROL 1 — pin model identity and version (revision + fingerprint; H4 lives here)
CONTROL 2 — pin prompt / rendered input (bytes + template version; Ch19)
CONTROL 3 — pin generation configuration (temperature, top-p, max tokens)
CONTROL 4 — pin seed where the runtime exposes one (global AND per-request)
CONTROL 5 — disable sampling where the question allows (temperature 0 narrows; it does not guarantee)
CONTROL 6 — pin runtime / serving configuration (flags, deterministic mode, prefix caching on/off)
CONTROL 7 — control batching / concurrency (solo vs mixed batch, off-peak reruns)
CONTROL 8 — pin hardware and deterministic-kernel settings (GPU type, TP degree, deterministic flags)

Deterministic guarantees are implementation/runtime controls, never universal model properties: batch-invariant paths exist on some stacks and are unavailable on others (quantized/MoE gaps), per-request seeds control the draw given logits but not the logits, and low-traffic single-request evals understate production variance because the batch is the variable.

The method: seed sweeps and distribution reads

1. Pin sampling as code. Temperature, top-p, max-tokens, seed (or seed-sweep plan), and the serving revision logged and hashed into the bundle. Anything left at “default” is recorded as UNKNOWN-by-default, not zero.
2. Run the N-trial series. Baseline: N≥20 trials, seeds swept and paired across arms (same seed set in baseline and every probe — paired differences are free variance reduction), outputs classified by the fixed signature (correct citation + amount / wrong citation / truncated / other). Report pass@1 rate with the raw counts — “14/20” before any percentage, because n=20 percentages pretend at precision the trial count never earned. There is no universal N: N≥20 is a floor for shape visibility (bimodality shows at twenty draws where it hides at three), probes run ×10 as directional-move detectors for forecasted effects ≥0.3, and anything needing a named confidence width (rate estimation, rare-tail hunts, intervention comparison) sizes N by power analysis, not ritual — small-N discrete rates are high-variance, and a 17/20 floor is a shape-gate, not a reliability claim.
3. Read the shape, then probe it. (a) Deterministic probe: temperature 0 (as implemented on the reader’s endpoint), same seeds ×10 — prediction if H1: collapse to one mode; (b) Top-p probe: narrowed nucleus on the same input — prediction if H1-variant: tail errors vanish; (c) Bimodality check: if two modes persist at deterministic settings, H2 stands — the system holds both answers and no sampling fix rescues it; repair is input, selection/verification, or weights (a verifier or reranker can ship H2 cases without touching weights). H3 is suspected only if identical nominal params produce different distributions across serving paths — then the plumbing, not the params, is the suspect. (d) Revision check: same bundle, same seeds, two revisions — prediction if H4: the paired delta survives even with the deterministic runtime on. Without this arm, drift misreads as plumbing.
4. Report pass@k correctly. pass@k answers “does any-of-k contain success” (selection ceiling); pass@1 answers “does the shipped draw succeed.” Quote both with N, or quote neither. Substituting one for the other is a category error, not a rounding error: an agent succeeding once in 20 retries can post a high pass@20 while remaining operationally unreliable, and pass@k must use the unbiased estimator with N stated, never the naive small-n fraction.

    flowchart TD
    P["pin sampling as code; run the deterministic-controls ladder"] --> N["baseline N>=20, seeds paired across arms, classify by the fixed signature"]
    N --> SH{"histogram shape?"}
    SH -->|"two persistent modes"| H2["H2: bimodal competence — no sampling fix; repair input, selector / verifier, or weights"]
    SH -->|"one mode + spread, or long wrong tail"| D["probe: temp 0 and narrowed top-p, same seeds x10"]
    D --> DD{"collapses to one correct mode?"}
    DD -->|yes| H1["H1: sampling spread — pin params, gate pass@1 in CI"]
    DD -->|no| RV["revision arm: same bundle + seeds, two revisions, deterministic runtime on"]
    RV --> RD{"paired delta survives deterministic-on?"}
    RD -->|yes| H4["H4: silent version / infra drift"]
    RD -->|no| H3["H3: serving plumbing — batch shape, TP order, speculative decoding"]
  

Determinism is achievable where you pay for it — batch-invariant stacks assert single-unique outputs across batch modes today — so this chapter reads as “solved where you pay for it,” never “impossible.” Most temp-0 instability in practice is a few flips at a few near-tie positions, which is exactly why semantic-cluster rates stay calmer than text-identity rates.

# distribution debugging (the object is the histogram, not the draw)
base = freeze(model_rev="rev-A", input_bytes=assembled,
              params={"temperature": 0.7, "top_p": 0.9, "max_tokens": 800})
baseline = run(base, seeds=range(20))          # MEASUREMENT: classify each output
print("pass@1:", rate(baseline), "modes:", modes(baseline))  # raw counts first
det = run(base.with_params({"temperature": 0}), seeds=range(10))  # H1 probe
narrow = run(base.with_params({"temperature": 0.7, "top_p": 0.5}), seeds=range(10))  # tail probe
# FORECAST: H1: det collapses to one mode >=9/10; H2: two modes persist even det;
# H3: same seeds, solo vs mixed batch -> paired rate delta CI excludes 0, closes under deterministic runtime.
# H4: same seeds across two revisions -> paired delta CI excludes 0 even deterministic-on.
print("pass@k (selection ceiling, k=10):", pass_at_k(baseline, k=10), "N=20 -- NOT shipped reliability")
print("paired delta (same seeds, baseline vs probe):", paired_delta_ci(baseline, probe))
# The only interval this chapter ever prints: a failure-rate delta on paired same-seed series.
# Full-distribution distances (KL/JS/Wasserstein) are explicitly out of scope here —
# they need logprobs, large N, and a reference the debugger doesn't have.

OBSERVATION (constructed illustration, not a measured run): baseline 12/20 correct-citation (bimodal: 12× 4.2-correct, 8× general-policy); deterministic probe 9/10 single-mode correct; narrowed top-p 8/10. UPDATED BELIEF: H1 supported for this fixture — spread, not bimodal competence; H3 exonerated here (serving path held constant). Deterministic settings are a measured result on this endpoint, not a cited guarantee. INFERENCE: pin deterministic params for this path + add the N-trial distribution to CI; the Tuesday-flicker ticket class closes by construction, not by luck.

Note what N=20 does and does not buy. It buys shape visibility — bimodality shows at twenty draws where it hides at three — and a pass@1 rate honest enough to gate on with a floor (“ship only above 17/20”). It does not buy a reliability certificate: the rate describes this input, this revision, this path. A second input needs its own series, which is why the classifier and harness are pinned as reusable assets rather than one-off scripts.

Research lineage: the shape, the tail, and the plumbing

pass@k has a precise definition, and it is not reliability. Chen and colleagues introduced pass@k with an unbiased estimator precisely because the naive “fraction of problems where any of k samples passed” is high-variance and optimistically biased at small sample counts (Chen et al., 2021). Report it with the estimator and the sample count n, label it a selection ceiling, and never let it stand in for the shipped pass@1.

The tail is the enemy, and top-p is the tool for it. Holtzman and colleagues showed that the low-probability tail — tens of thousands of tokens with tiny individual probability but large aggregate mass — is what produces incoherent or off-topic generations, and that lowering temperature reshapes the distribution without actually suppressing that tail. Nucleus (top-p) sampling truncates it directly (Holtzman et al., 2020). Newer levers share the debugging role: locally-typical sampling truncates by entropy-deviation (Meister et al., 2023), and min-p sets a dynamic threshold scaled to the top token that holds coherence at high temperature (Nguyen et al., 2025; adopted by major frameworks). Same role, newer tools. This is why the chapter’s shape probe varies truncation, not just temperature: tail errors and mode-selection errors need different levers. And mean, mode, and tail are different debugging targets — excellent average quality with a stable top response can coexist with rare catastrophic tail failures, which is what motivates the retry, hallucination-tail, and incident analyses later.

H3 has a corroborated mechanism, an unreplicated rate, and two competing prescriptions. The batch-invariance failure He describes is the mechanism behind “same nominal params, different distribution across serving paths” — and it is no longer one lab’s blog claim: Yuan et al. give a peer-reviewed characterization of the same precision / reduction-order / batch-size sources of inference nondeterminism (Yuan et al., NeurIPS 2025), SGLang ships a deterministic-inference mode built on batch-invariant operations (50-sample reproduction of the fix’s effect: 6 unique outputs down to 1; overhead 34% with CUDA graphs versus ~61% unoptimized), vLLM ships a batch-invariant execution path, and Microsoft’s LLM-42 preprint confirms the diagnosis while proposing a different remedy — decode → verify → rollback, with per-request selective determinism instead of invariant kernels (He, Thinking Machines Lab, 2025; Gond et al., Microsoft Research, 2026). Agreement about the mechanism does not imply agreement about the engineering intervention: invariant kernels trade throughput and hardware coverage for exactness (unavailable on some quantized/MoE stacks), while verify-and-rollback trades runtime complexity for selectivity. The dossier classification is PARTIALLY CORROBORATED — mechanism corroborated, 80/1,000 rate unreplicated — with two disclosures the chapter owes the reader: the headline fix demonstration depended partly on unreleased code, and no independent rerun of the exact rate exists to date. The debugging takeaway stands, sharpened: if a distribution shifts when nothing in your bundle changed, suspect the server’s batching and a silent revision — H3 and H4 are competing suspects, separated by the paired-revision probe in Lab 21. No later chapter may generalize the rate into a law of deterministic inference.

The distribution can also be used, not just debugged. Self-consistency — sample many chains, take the majority answer — is the constructive reading of a multi-modal histogram: when the correct mode is the largest, aggregation recovers it (Wang et al., 2023). Measured boundary: selector-based gains plateau around ~100 samples while oracle coverage keeps climbing — past the plateau, more samples buy coverage only with a better verifier (Brown et al., 2024). If your H2 bimodality has the right answer as the dominant mode, majority-vote selection is a mitigation; if the wrong mode dominates, it is not.

Lab 21: the N-run distribution

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own trial series.

Setup. Take one flickering case (or inject spread: raise temperature on a passing deterministic case). Pin the bundle including sampling params. Fix the output classifier before running (correct / wrong-citation / truncated / other).

Task.

1. Write H1/H2/H3/H4 with distributional FORECASTs before sweeping (e.g., “H1: deterministic probe single-mode ≥9/10; H2: two modes persist ≥3/10 each at deterministic; H3: solo-vs-mixed batch moves the paired rate with identical params and seeds, closing under the deterministic runtime; H4: paired seeds across two revisions move the rate even deterministic-on”).
2. Independent variable per series: sampling params (H1), truncation (H1-tail), serving path or batch shape (H3), revision (H4). Controlled variables: bytes, classifier, and (except in their own arm) revision, path, batch shape — frozen, seeds paired across all arms.
3. Run baseline N≥20 (paired seed sweep) + deterministic probe ×10 + shape probe ×10 + revision arm ×10 (same seeds, second revision or fingerprint). Record OBSERVATION (per-trial hash + signature + result verbatim) and UPDATED BELIEF. Any single-draw verdict is UNKNOWN by rule. Under a deterministic runtime, also record unique-hash counts — single-unique across the arm is the shipped gate.
4. Pin the sampling configuration that the distribution justifies — or file the bimodality finding (H2) with its two-mode table and route to input, selection/verification, or weights repair.

Success criterion. A completed per-trial table (N≥20 + probes + revision arm) matching one pre-written distributional pattern, with pass@1 (counts), pass@k (labeled ceiling, unbiased estimator, N stated), and the paired-delta CI reported. A single passing retry is explicitly not completion.

Companion tool: N-Run Distribution Explorer

What it accepts: run records (per-trial hash, signature, result, latency), model/runtime identity with revision fingerprints, generation settings, the behavioral-signature definition, and batch-shape/position metadata. What it performs: it renders the outcome distribution with signature counts, computes failure rate and pass@1 (counts + rate) plus pass@k (labeled selection-ceiling, unbiased estimator), compares conditions on paired same-seed deltas with a confidence interval, and raises unexpected-variation warnings (a revision-fingerprint change mid-series, a batch-shape skew, a signature the classifier never named). What it can establish: which distributional story holds for this input under these params — spread, bimodality, plumbing, or drift — and the measured pass@1 rate at the examined N only. Variation detection is not causal diagnosis; the tool never claims a root cause was automatically identified. What it cannot establish: reliability beyond the fixture, cross-endpoint determinism (serving behavior is changeable and locally measured), or that pass@k describes shipped behavior — it labels pass@k a ceiling on every render. It never treats one draw, agreement across paraphrases, or a confident tone as distributional evidence. How its output changes your next action: an H1 conviction routes to pinned sampling + distribution CI gates; H2 routes back to Chapters 18–20 (the input needs repair — both modes are honestly held); H3 routes to serving-path investigation with params held constant.

Paper form, sufficient for this chapter:

PARAMS: temp ___ | top_p ___ | max_tok ___ | seeds (paired) ___ | model rev+fingerprint ___ | serving rev ___ (defaults: none unlogged)
BATCH: shape ___ | position ___ | concurrency ___ | deterministic flag ___
BASELINE (N=20): sig-A __ | sig-B __ | trunc __ | other __  => pass@1 __/20
DET ×10: ___  NARROW/PATH ×10: ___  REVISION ×10: ___  pass@10 (ceiling, unbiased, N=20): ___
PAIRED DELTA (probe vs baseline): ___ [95% CI ___]
CONVICTION: H1 / H2 / H3 / H4 / UNKNOWN   PINNED: ___

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. The histogram discipline precedes any automation.

Reusable procedure: every flicker gets this series

1. Classify first — output classes fixed before any trial.
2. Sweep the baseline — N≥20, seeds recorded, counts raw.
3. Probe deterministic — same input, deterministic params ×10.
4. Probe shape — narrow top-p or alternate path ×10 against H2/H3.
5. Pin and gate — sampling config + pass@1 floor in CI.

Failure modes

• Retry-as-verdict. One passing re-run closing the ticket. Draws are not verdicts; distributions are.
• pass@k as reliability. Quoting any-of-many success for one-draw shipping. Label the ceiling or mislead the roadmap.
• Unlogged sampling. Temperature and seed absent from the bundle. The experiment is then unrepeatable and every comparison UNKNOWN.
• Cross-endpoint determinism claims. “Temperature 0 is deterministic everywhere” as fact. Determinism is a local MEASUREMENT on a named endpoint and revision, valid until the next deploy — attribute and re-measure.
• Bimodality patched with temperature. Narrowing spread around two honestly-held answers and calling one setting “fixed.” Two modes mean two attractors; no sampling fix rescues them — repair is input, selection/verification, or weights (Chapters 19–20, 22–24).
• Classifier-after-looking. Writing the output classes after seeing the trial series. Post-hoc classes fit the noise — fix the rubric first or the histogram is a Rorschach test.
• Borrowed N. Citing another team’s trial count as your reliability. Distributions are local to input, revision, and path; imported rates are hearsay.
• Borrowed determinism. Importing another team’s temp-0 stability without re-measuring under local load. Replay is envelope-local; their calm batch is not your production batch.
• Drift misread as plumbing. An H3 conviction without the paired-revision arm. Same-symptom, different owner — run H4 before blaming the batcher.
• Hardware-gated guarantees. Invoking deterministic flags on stacks where they are unavailable (quantized/MoE paths). The control is absent, not zero — record UNKNOWN, not “deterministic.”

Limits, per contract: one distribution describes one input under one param set, revision, and serving path at N stated; it does not certify the fixture, does not transfer across endpoints, and expires on any param/revision/path change. A deterministic runtime does not make the model correct — only narrower. Estimated rates move with prompt, model, hardware, serving system, and version; a low observed failure rate is not proof of absence. UNKNOWN wherever N<10, seeds unrecorded, or the classifier was written after looking.

References

• Mark Chen, Jerry Tworek, Heewoo Jun, et al. Evaluating Large Language Models Trained on Code. arXiv:2107.03374, 2021 (defines pass@k and its unbiased estimator). https://arxiv.org/abs/2107.03374
• Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The Curious Case of Neural Text Degeneration. International Conference on Learning Representations (ICLR), 2020. https://arxiv.org/abs/1904.09751
• Horace He and the Thinking Machines Lab. Defeating Nondeterminism in LLM Inference. Thinking Machines Lab Blog Connectionism, 10 Sep 2025. https://doi.org/10.64434/tml.20250910 (lab report, not peer-reviewed; mechanism now corroborated by the peer-reviewed characterization below and by framework adoption; 80/1,000 rate unreplicated — see text).
• Jiayi Yuan, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference. Advances in Neural Information Processing Systems 38 (NeurIPS), 2025 (arXiv:2506.09501). https://arxiv.org/abs/2506.09501 (peer-reviewed: precision / reduction-order / batch-size sources; LayerCast mitigation).
• Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-Consistency Improves Chain of Thought Reasoning in Language Models. International Conference on Learning Representations (ICLR), 2023. https://openreview.net/forum?id=1PL1NIMMrw
• Brown, Juravsky, Ehrlich, Clark, Le, Ré, and Mirhoseini. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. arXiv:2407.21787, 2024. https://arxiv.org/abs/2407.21787 (coverage ≡ pass@k; selection plateaus ~100 samples while oracle coverage climbs; given names unverified — surnames as listed).
• Miller. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640, 2024. https://arxiv.org/abs/2411.00640 (paired differences, resampling rule, power analysis; never tune temperature for variance; author given name unverified).
• Madaan, Singh, Schaeffer, Poulton, Koyejo, Stenetorp, Narang, and Hupkes. Quantifying Variance in Evaluation Benchmarks. Advances in Neural Information Processing Systems 37 (NeurIPS), 2024. (small-benchmark variance; bootstrapped CIs; continuous metrics carry higher signal; given names unverified — surnames as listed).
• Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. Locally Typical Sampling. Transactions of the Association for Computational Linguistics (TACL), 2023. (entropy-anchored truncation alternative to nucleus sampling).
• Nguyen, Baker, Neo, Roush, Kirsch, and Shwartz-Ziv. Turning Up the Heat: Min-p Sampling. International Conference on Learning Representations (ICLR), 2025. (dynamic threshold holding coherence at high temperature; framework-adopted; given names unverified — surnames as listed).
• Gond, Kamath, Ramachandran, and colleagues (Microsoft Research). LLM-42: Enabling Determinism in LLM Inference with Verified Speculation. arXiv:2601.17768, 2026. https://arxiv.org/abs/2601.17768 (confirms the batch-non-invariance diagnosis; contests the prescription with decode → verify → rollback; given names unverified — surnames as listed).
• Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting Hallucinations in LLMs Using Semantic Entropy. Nature 630, 2024. (meaning-cluster entropy as confabulation detector — detection, not localization; the measurement behind signature-level counting).

Debugging Checklist

• Output classifier fixed before any trial?
• Sampling params (temp, top-p, max-tokens, seeds, serving rev) pinned in the bundle?
• Baseline N≥20 with seed sweep; raw counts recorded?
• H1/H2/H3/H4 distributional FORECASTs written before probing (H4: paired-revision arm)?
• Deterministic + shape probes run ×10 each?
• pass@1 (counts) and pass@k (labeled ceiling) reported separately?
• Classifier written before the first trial (not after looking)?
• Serving path + revision logged alongside every series (no borrowed rates)?
• No single-draw verdict recorded anywhere?

What This Chapter Established

• Distributions as the debugging object: N-trial series with paired seeds, shape reads (spread/bimodal/plumbing/drift), pass@k read strictly as selection ceiling via the unbiased estimator — demonstrated on the flickering refund case as H1 — constructed illustration, no measured runs claimed.
• Behavioral signature as a formal concept (task-relevant projection with the byte → token → semantic-cluster → task-success hierarchy), the deterministic-controls hierarchy (eight rungs; determinism as implementation control, never model property), and question-dependent N (shape-visibility floors, paired deltas, power-sized rate claims).
• Lab 21 as a proposed per-trial record the reader executes, now with the H4 paired-revision arm; the N-Run Distribution Explorer contract upgraded to run records, identities, signature definitions, and paired-CI condition comparison (variation detection, never automatic causal diagnosis).
• What was NOT proved: any cross-endpoint determinism claim, any fixture-wide reliability rate, any universal N, or anything about the weights beyond the measured shape on this input.
• Research grounding: the He batch-non-invariance mechanism is corroborated by shipped framework fixes with the 80/1,000 rate kept as a bounded existence proof (PARTIALLY CORROBORATED dossier verdict); LLM-42 contests the prescription, not the diagnosis; pass@k/selection-plateau (Chen; Brown), tail truncation old and new (Holtzman; Meister; min-p), eval-variance rules (Miller; Madaan), and semantic entropy as detector-not-localizer (Farquhar) ground the method; flaky-test and statistical-debugging lineage names what is inherited versus genuinely new.
• Forward link: distributions describe what the system does across draws, grouped by signature. They never say where inside the computation doubt concentrates — and shape now routes the question: flat tail to truncation territory, bimodality away from entropy signals. The next chapter opens the weak interior signals — and the strict limits on reading them.

Next

The distribution is now honest: measured, shaped, pinned. But one question survives every histogram — when the system goes wrong, is there any interior signal that locates the doubt before the wrong words ship? Logprobs dip, entropy spikes, attention lights up. The next chapter treats those signals as what they are: leads for triage, never explanations.