Part VIII — Building the AI Debugger

Three hypotheses, nine runs, one lesson

Chapter 47 mapped the space; this chapter prices the search. A practitioner holds three live hypotheses for a RAG fabrication — stale snapshot (H1), reranker demotion (H2), prompt override (H3) — and runs them naively: re-index (expensive, hours), retrain the reranker (days), rewrite the prompt five ways (scattershot). Nine runs later H1 is confirmed by accident — the re-index fixed it — but the prompt rewrites also “helped” once each, the reranker retrain changed scores without changing outcomes, and nobody can say which run proved what because two variables moved per run and predictions were written after seeing outputs.

OBSERVATION: nine interventions executed, multiple variables changed per run (re-index plus prompt edits together), predictions recorded post-hoc, trial counts of one per condition. HYPOTHESIS H1/H2/H3 (carried from Ch47): each predicts a distinct signature — but the run log cannot assign outcomes to hypotheses because runs were not discriminating. INFERENCE: none yet — the same nine runs, redesigned as three single-variable discriminating experiments with pre-written predictions, would have decided the space; as executed they established effort, not cause.

This chapter’s question: how do we design each experiment for maximum information per run — splitting the hypothesis space fastest with the fewest single-variable interventions?

Why “try the most likely fix first” fails first

The obvious move — intervening on the favorite hypothesis immediately — fails because confirmatory runs teach little and confound much. Five defects hide behind fix-first experimentation:

1. Multi-variable runs. Changing the index and the prompt together: any outcome is uninterpretable. One variable per run or the run teaches nothing (Ch01’s rule, now priced in hours).
2. Confirmatory-only design. Testing H1 in a way H1 passes under every hypothesis (“re-run and see if it looks better”). Runs must be able to fail each hypothesis distinctly.
3. Prediction-after-outcome. Writing “as expected” after seeing the result. Post-hoc predictions are storytelling with timestamps missing (I-4 fails).
4. Single-trial verdicts on varying systems. One seed, one run, one conviction. Nondeterministic steps need ≥3 trials per condition before any branch closes.
5. No factorial discipline. Testing H3’s five prompt rewrites while H1/H2 stay uncontrolled, so prompt “effects” are retrieval noise. Factors vary one at a time or in a planned factorial — never by drift.

OPINION: an experiment budget spent fix-first buys motion; the same budget spent discrimination-first buys decisions. Runs are currency — spend each on eliminating branches, not on comforting favorites.

The mental model: each run is a question that splits the space. A discriminating experiment is designed so its possible outcomes assign differently across live hypotheses: outcome A exonerates H1 and keeps H2, outcome B the reverse. Information per run is the number of branches a single outcome closes. The best next run is the one whose outcomes partition the live set most evenly at the lowest cost — cheap, decisive splits before expensive, confirmatory ones.

This is Platt’s strong inference, made concrete — itself a sharpening of Chamberlin’s older method of multiple working hypotheses, which urged holding several explanations in mind at once precisely so that no single one captures the investigator’s loyalty before the evidence is in (Chamberlin, 1897). Platt argued that fast-moving sciences all run the same loop: devise alternative hypotheses, devise a crucial experiment that will exclude one or more of them, perform it cleanly, then recycle on what remains — a “conditional inductive tree” like the branching table for qualitative chemical analysis (Platt, 1964). Chapter 47 built the tree; this chapter designs each crucial experiment. Formally, “branches closed per run” is a discrete stand-in for the expected information gain of an experiment (Lindley, 1956); the recycle-after-every-outcome loop is its sequential form, which modern Bayesian optimal experimental design formalizes. Strong inference has documented limits — it presumes the true cause is among the enumerated hypotheses and that a clean crucial experiment exists, and it under-weights prior evidence in deciding which experiment is genuinely crucial (a retrospective at fifty years and its critics — Fudge, 2014). This chapter is bracketed by the two that answer that: Chapter 47’s coverage discipline (is the true cause even on the list?) and Chapter 49’s survival bar (a branch can survive every split and still be wrong).

The method: discriminating power with factorial discipline

Design every run with four fields filled before execution:

1. Single variable. Exactly one factor changes (evidence set, wording, seed, k, version pin); all else frozen to bundle hashes. Two changes → two runs, no bundling.
2. Pre-written divergent predictions. Each live hypothesis predicts a distinct observable for this run. If two hypotheses predict the same outcome, the run cannot split them — redesign.
3. Trial count set by variance. Deterministic steps: 1 trial suffices with hashes verified. Nondeterministic steps (temperature > 0, unseeded sampling, racing tools): ≥3 trials per condition, outcome reported as stability (3/3, 2/3) never as a single anecdote.
4. Kill-first ordering. Run the cheapest intervention that exonerates the most branches first. Presence-checks (is the chunk in the list?) precede re-indexes; k-expansion precedes retrains; wording-swaps precede architecture changes.

    flowchart TD
    B["live branches from the Ch47 space (exonerated ones struck, deciding run cited)"] --> D["design one run: exactly one variable changes, all else frozen to bundle hashes"]
    D --> P{"each live hypothesis predicts a DISTINCT observable for this run?"}
    P -->|no| RD["redesign — a run that can't split them teaches nothing"]
    P -->|yes| T["set trial count by variance: 1 if deterministic, >=3 if sampling / racing"]
    T --> O["order runs kill-first by (branches closed / cost): list inspections before re-indexes, k-expansion before retrains"]
    O --> RUN["run the cheapest most-splitting series; record branches closed per outcome"]
    RUN --> RT{"more than one live branch remains?"}
    RT -->|yes| RETAB["re-table before the next run (Platt's recycle step)"]
    RETAB --> D
    RT -->|"one branch"| CONF["route to Ch49 confirmation (small factorial over the surviving factors)"]
  

DISCRIMINATION TABLE (worked sketch; constructed, not a measured run):
Symptom: fabricated citation. Live: H1 stale / H2 demotion / H3 override.
RUN A (cheap): inspect ranked list for gold chunk (no execution).
  H1 predicts ABSENT | H2 predicts PRESENT-below-k | H3 predicts either.
  Cost ~0 → splits H1 vs H2 outright. RUN FIRST.
RUN B (cheap): k-expansion 5→50, all else frozen, ×3 trials.
  H1 predicts no change (absent stays absent) | H2 predicts citation returns.
  Splits H1/H2 behaviorally if A was ambiguous. RUN SECOND.
RUN C (cheap): wording-swap with evidence frozen, ×3 trials.
  H3 predicts citation returns | H1/H2 predict unchanged fabrication.
  Splits instruction vs evidence layers. RUN THIRD.
RULE: A→B→C costs three cheap series; the naive re-index-first plan
cost one expensive run that H2/H3 could neither confirm nor kill.

OBSERVATION (constructed illustration, not a measured run): Run A shows the gold chunk absent; Run B changes nothing ×3; Run C changes nothing ×3. UPDATED BELIEF: H1 supported (absence + B/C negatives converge); H2 exonerated here (expansion cannot restore what absence explains); H3 exonerated here (wording inert with evidence frozen) — three cheap series, one surviving branch, predictions pre-dated.

No reranker score delta, no confidence shift, no agreement between repeated assistant judgments, no single improved anecdote, and no downstream symptom quieting substitutes for a pre-registered split with trial counts. Tables, not impressions.

Example: the factorial the naive plan skipped

The naive nine runs changed index + prompt + k across overlapping trials. The discriminating redesign holds a factorial: evidence state (stale/fresh) × wording (orig/swap) × k (5/50), varying one factor per series with the others frozen, ≥3 trials per cell where sampling varies. The table fills in four series instead of nine, and the H1 cell (fresh + orig + 5 → citation returns ×3) with the stale-cell contrast is the intervention-effect confirmation Chapter 49 will formalize. The saving is not just runs — it is interpretability: every cell’s outcome is assignable to exactly one factor.

Research lineage: one factor at a time, then a factorial

OFAT is for attribution; a factorial is for efficiency and interactions. Fisher’s argument against pure one-factor-at-a-time experimentation was that a factorial design learns as much about many factors in the same number of trials and reveals interactions that OFAT structurally cannot — the classic example is three treatments whose combination multiplied a bearing’s lifespan fivefold, an effect decades of one-at-a-time testing missed (Fisher, 1935). This chapter’s rule is not “OFAT always.” It is: use single-variable runs for the discriminating phase, where the goal is clean attribution of which branch an outcome closes; then, in the confirmation phase (Chapter 49), run a small factorial over the surviving factors — evidence state × wording × k — because that is where an interaction (a wording that only matters under stale evidence) would hide.

The recycle step is the discipline people skip. Platt’s loop does not end at “perform the experiment”; it ends at “recycle on what remains.” The chapter’s “re-table after every outcome before the next run” is that step. A team that designs one discrimination table and then executes it front-to-back without re-tabling has stopped doing strong inference after the first branch.

Lab 48: redesign one naive run into a discriminating series (proposed)

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own design table.

Setup. Take one live symptom with ≥2 competing hypotheses and one naive run you already performed or planned. The design regime (naive fix-first vs. this chapter’s discrimination table) is the independent variable; symptom, bundle, and system are controlled.

Task.

1. Before redesigning, write H1/H2/H3 about the regimes with distinct predicted information outcomes: H1: “the table’s first run splits ≥2 branches at lower cost than the naive run”; H2: “both designs decide equally fast”; H3: “the naive run decides faster (discrimination overhead wastes runs).”
2. Fill the discrimination table: one variable per run, divergent pre-predictions per live hypothesis, trial counts by variance, kill-first order.
3. Execute the table’s first series only (≥3 trials if nondeterministic); record branch closures per run.

Success criterion. A filled discrimination table with pre-dated predictions plus an execution log showing branches-closed-per-run. A fix applied without the table is explicitly not completion.

Companion tool: Hypothesis Experiment Designer

What it accepts: the enumerated hypothesis space (Ch47), the bundle manifest, and the per-run cost/variance notes. What it performs: it enforces one variable per run, requires divergent pre-predictions per live hypothesis, sets trial counts (1 deterministic / ≥3 nondeterministic), orders runs kill-first by (branches-closed ÷ cost), and emits the discrimination table above. What it can establish: which run to do next and what each possible outcome would close — for the examined space only. What it cannot establish: the cause, cross-space optimality, or future cost estimates. It never treats scores, confidences, agreement, single outcomes, or symptom relief as design inputs. How its output changes your next action: table emitted → execute the first row with predictions locked; rows with overlapping predictions → redesign before running; no cheap splits remain → escalate to the expensive decisive run with full trial discipline.

Paper form, sufficient for this chapter:

Space: ___ (live H ___)  Bundle: ___
RUN ___: vary ___ freeze ___ trials ___ cost ___
  H1 predicts ___ | H2 predicts ___ | H3 predicts ___
  closes: ___ on outcome A / ___ on outcome B
ORDER (branches/cost): ___ → ___ → ___  PREDICTIONS DATED: ___

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. Split, don’t splash.

Reusable procedure: price every run in branches closed

1. List live branches — from the Ch47 space, exonerated ones struck with their deciding run cited.
2. Design one variable — single factor, rest frozen to hashes.
3. Pre-write divergences — every live H predicts distinctly, dated before running.
4. Set trials by variance — 1 where deterministic, ≥3 where sampling or racing varies.
5. Kill-first order — cheapest most-splitting run next; re-table after every outcome.

Failure modes

• Favorite-first runs. Testing H1 because it is liked. Liked is not discriminating; splits order runs.
• Bundled interventions. Two changes, one run, zero attribution. Unbundle or discard.
• Post-dated predictions. “Predicted” after observed. Undated predictions are inadmissible (I-4).
• Single-trial closure. One varying run exonerating a branch. Variance obligates repetition.
• Score-watching. Tracking reranker scores across runs as the outcome. Scores are not intervention effects; behavior deltas are.
• Run-until-relief. Stopping when the symptom quiets rather than when branches close. Relief ends toil; closure ends diagnosis.

Limits, per contract: one table covers one space under one bundle; it warrants run-efficiency, not cause; version/index/prompt changes after tabling void the frozen columns until re-tabled. UNKNOWN wherever variables moved together or predictions post-date outcomes.

References

• John R. Platt. Strong Inference. Science 146(3642), 1964, pp. 347–353. https://www.science.org/doi/10.1126/science.146.3642.347
• Ronald A. Fisher. The Design of Experiments. Oliver and Boyd, Edinburgh, 1935.
• Dennis V. Lindley. On a Measure of the Information Provided by an Experiment. The Annals of Mathematical Statistics 27(4), 1956, pp. 986–1005. https://doi.org/10.1214/aoms/1177728069
• T. C. Chamberlin. The Method of Multiple Working Hypotheses. Journal of Geology 5(8), 1897, pp. 837–848 (reprinted Science 148, 1965). https://doi.org/10.1126/science.148.3671.754
• Douglas S. Fudge. Fifty Years of J. R. Platt’s Strong Inference. Journal of Experimental Biology 217(8), 2014, pp. 1202–1204. https://journals.biologists.com/jeb/article/217/8/1202

Debugging Checklist

• Live branches listed with exonerated ones struck and cited?
• Exactly one variable per designed run (rest hash-frozen)?
• Every live hypothesis predicts a distinct outcome, pre-dated?
• Trial counts set by variance (≥3 where nondeterministic)?
• Runs ordered cheapest-most-splitting-first?
• No scores/confidences/agreements/single anecdotes as outcomes?
• Branches-closed-per-run recorded after each execution?
• Table re-issued after every outcome before the next run?

What This Chapter Established

• Discriminating experiment design: single-variable runs with divergent pre-predictions, variance-set trial counts, and kill-first ordering priced in branches-closed-per-run — demonstrated on the constructed A→B→C table, no measured runs claimed.
• The run-as-space-split mental model with factorial discipline (one factor per series, interpretable cells).
• Lab 48 as a proposed naive-to-discriminating redesign record the reader executes; the Hypothesis Experiment Designer contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: any cause, any cross-space optimality claim, or any cost generality. One table designed; nothing concluded.
• Research grounding: this is Platt’s strong inference (sharpening Chamberlin’s older method of multiple working hypotheses) — alternative hypotheses, a crucial experiment that excludes one or more, a clean result, then recycle (the conditional inductive tree); “branches closed per run” is a discrete expected-information-gain (Lindley), and the recycle loop is its sequential form (modern Bayesian OED); single-variable runs give clean attribution in the discriminating phase, but the confirmation phase (Ch 49) wants a small factorial because interactions hide from one-factor-at-a-time testing (Fisher); the recycle / re-table step is the one teams skip; and strong inference’s known limits (it presumes a complete hypothesis set and a clean crucial experiment — Fudge and its critics) are answered by Ch 47’s coverage and Ch 49’s survival bar.

Next

Cheap splits decide most spaces — but deciding is not verifying. A branch can survive every split yet still be wrong (shared confounder, misfrozen variable, mistaken signature). Survival demands a higher bar before anyone signs a diagnosis. Chapter 49, “How Do You Know the Diagnosis Is Right?,” sets that verification standard — prediction-match ledgers and independence of verification evidence.