Part I — Debugging From First Principles

The puzzle: two fixes, one bug

A team owns a small billing function. Invoices above €1,000 get a 10% discount. One Monday, a €1,200 invoice goes out at full price. Two developers investigate independently.

Developer A reads the code, spots a suspicious comparison, flips < to <= somewhere that “looks wrong,” reruns the failing invoice, and sees €1,080. Fixed. She closes the ticket.

Developer B writes down the intended behavior first: “orders with total > 1000 receive total * 0.9.” She reproduces the failure with a minimal script, records the actual intermediate values, finds where they first depart from intent, tests one change, reruns the reproduction plus edge cases, and adds a regression test. Her fix is in a different line than A’s.

Both developers saw the symptom disappear. Only one of them debugged.

This chapter defines the difference. Debugging is the construction of the smallest evidence-backed causal account that predicts the failure and its reversal — pursued, inside a pinned input × version × oracle envelope, by finding the earliest transition that survives forward and reverse intervention; outside the envelope, by difference → relevance → support, capped honestly at PROVISIONAL or INCONCLUSIVE. Fixing makes a symptom go away. Debugging establishes where execution first departed from intent, why, and how you would catch it next time.

DOMAIN ENVELOPE: one input (or minimized class), one code version, one environment, an oracle whose verdicts are stable under repetition, a writable intent, and a state the debugger can observe and set. Inside the envelope, seek the earliest divergence surviving forward + reverse intervention. Outside any element of the envelope, log the missing element as the reason and cap at PROVISIONAL (multi-cause / irreversible) or INCONCLUSIVE (no clean intervention).

A “root cause” in this book is that account scoped to its envelope — not an omniscient reconstruction.

OBSERVATION: two edits can both make one failing input pass while only one addresses the cause. INFERENCE: symptom relief is not a diagnosis.

That distinction carries the entire book — from Python tracebacks to hallucinations to agent trajectories.

Why “it works now” is insufficient

Consider the actual code, simplified:

def invoice_total(items, discount_threshold=1000, discount_rate=0.10):
    subtotal = sum(i["price"] * i["qty"] for i in items)
    discount = 0.0
    if subtotal >= discount_threshold:  # intended: strictly greater than?
        discount = subtotal * discount_rate
    total = subtotal - discount
    tax = total * 0.20
    return round(total + tax, 2)

The failing order: two items at €600 each, qty=1. Expected (per spec total > 1000): subtotal 1200, discount 120, total 1080, tax 216, invoiced 1296. Observed: 1440.

Developer A “fixed” the tax rounding line. It happened to change this one output by coincidence of her test data — or she tested with different items where rounding masked the discount path. The details do not matter. What matters: she never established the divergence point, so she cannot distinguish these hypotheses:

• H1: the discount condition is wrong (boundary or comparison error).
• H2: the subtotal computation is wrong (bad input parsing, float error).
• H3: the discount is computed but dropped downstream (overwrite, wrong variable).

All three produce “wrong invoice amount.” Only evidence at intermediate points separates them — each predicts a different first wrong intermediate:

Read the intermediates in order and the first mismatch names the hypothesis; the final €1,440 names none of them.

This is the first discipline of the book:

1. Observation must be separable from explanation. “Invoice was €1,440, expected €1,296” is observation. “The comparison is off by one” is explanation. Never let the second overwrite the first.
2. Reproduction quality sets the ceiling on diagnosis quality. If you cannot retrigger the failure deterministically, every conclusion after that is speculation.
3. A diagnosis is incomplete until it predicts intervention outcomes. “I think it’s X” must cash out as: “if I change X and nothing else, Y will happen” — and then Y happens.

The debugging object: values, then everything else

Every failure in this book is a statement about a debugging object — the thing you inspect to find divergence. This book uses five, in order of increasing difficulty:

Values        — a single number, string, flag (this chapter)
State         — values evolving over time in a program
Distributions — outputs that vary across runs, data, seeds
Evidence      — retrieved / cited material a claim depends on
Trajectories  — sequences of tool calls, generations, actions

Chapter 1 works only with values: the subtotal is 1200 or it is not; the discount is 120 or it is not. That is deliberate. If you cannot isolate a wrong value in a deterministic five-line function, you have no chance against a RAG pipeline or an agent that fails once in twenty runs.

Routing by object at all is the expert move. Studies of expertise going back to Chi, Feltovich, and Glaser (1981) find that novices sort problems by surface features while experts sort by the deep structure that determines the method. “It’s slow,” “it hallucinated,” “the agent gave up” are surface features; a wrong value, an accumulated state, a distribution, an evidence chain, a trajectory are the deep-structure categories, and each one selects a different diagnostic method. Chapter 60 closes the book on this point; the whole book is training for it.

The book-wide rule already applies here:

Find the earliest causal divergence: the first difference on the path to the failure that moves the outcome when changed and nothing else is. A bare first difference is a lead; only the third ladder level earns the word cause. Outside the envelope above, the honest output is PROVISIONAL or INCONCLUSIVE, not a cause.

In the invoice example, that means checking subtotal, then discount, then total, then tax — in order — against intent, and stopping at the first mismatch. Everything downstream of that point is effect, not cause.

One refinement, which the research later in this chapter forces. “Earliest divergence” is too blunt if taken literally, because two executions can differ early for reasons that never touch the failure. The rule has three levels:

FIRST DIFFERENCE          any intermediate that differs from intent
        ↓                 → candidate for localization
FIRST RELEVANT DIFFERENCE a difference on the path to the failure
        ↓                 → supported hypothesis
FIRST CAUSAL DIVERGENCE   changing it, and nothing else, changes the outcome
                          → confirmed cause

The book’s working definition is the third line. A difference earns the word cause only when an intervention that changes it changes the failure as predicted. Everything above that line is a lead, not a verdict.

A related primitive, also from the research below, is the cause transition: the moment in execution where some variable ceases to be a failure cause and another begins — the cause set changes membership, and the code executed between the two points is the fix candidate. It is usually not the line that finally emits the wrong value, and not necessarily the line that contains the defect:

subtotal = 1200
        ↓
discount = 120
        ↓
discount = 0        ← cause transition: `discount` becomes failure-relevant here
        ↓
total = 1200
        ↓
wrong invoice

That distinction — failure location ≠ defect location ≠ cause transition — scales through the whole book. For a retrieval pipeline the cause transition might be “document dropped during context assembly”; for an agent, “tool observation never written to state.” The emitted symptom is always downstream of it.

The loop (inherited, not repeated)

The book-wide debugging loop is:

OBSERVE → REPRODUCE → MINIMIZE → LOCALIZE → HYPOTHESIZE → EXPERIMENT → VERIFY → PREVENT

This chapter does not ask you to memorize it. It asks you to understand why each step exists, using the invoice bug as the running example:

• OBSERVE. Capture intended vs. observed as precisely as possible. “Wrong total” is not an observation. “subtotal=1200 (intended 1200), discount=0.0 (intended 120), total=1200 (intended 1080)” is.
• REPRODUCE. Build a script that triggers the failure on demand, pinned to inputs, code version, and environment. If reproduction requires clicking through a UI five times, your diagnosis ceiling is already low.
• MINIMIZE. Shrink the reproduction to the smallest input that still fails. Two items at €600 is better than a 40-line order export. Minimization removes confounders — and, as the next section shows, it is itself a sequence of experiments.
• LOCALIZE. Walk the execution in order and mark the first divergence. This is Chapter 2’s full subject; here, note only that localization precedes confirmation — theorizing may start anywhere; confirmation may not. Recognition first where cues are valid; hypotheses first where they aren’t.
• HYPOTHESIZE. Generate competing hypotheses compatible with the evidence. Plural. If you have one hypothesis, you have a guess.
• EXPERIMENT. Change one variable, predict the outcome in advance, observe. A change without a prior prediction is tinkering.
• VERIFY. Rerun the reproduction, edge cases, and the surrounding test suite. One passing run is not verification (more in Chapter 3 on evidence).
• PREVENT. Convert the diagnosis into an artifact: a regression test, an assertion, a checklist entry. Prevention is not part of the definition of debugging; it is part of this book’s operational discipline — the step that converts a private diagnosis into a collective asset. A fix without prevention is a future recurrence.

Developer A did step 6 only. Developer B did all eight.

Minimization is an experiment, not cleanup

MINIMIZE is the step readers most often treat as tidying — shrink the repro so it is nicer to look at. The research reframes it. Delta debugging shrinks a failing input by repeatedly testing smaller variants: each variant that still fails removes a set of circumstances from suspicion. Minimization is a sequence of discriminating experiments whose byproduct happens to be a small test case.

Here is a teaching implementation — an educational simplification of ddmin, not a faithful one:

def minimize_failure(parts, still_fails):
    """Shrink `parts` to a near-minimal sublist for which still_fails(sublist) is True."""
    current = list(parts)
    granularity = 2
    while len(current) >= 2:
        chunk = max(1, len(current) // granularity)
        for start in range(0, len(current), chunk):
            candidate = current[:start] + current[start + chunk:]
            if candidate and still_fails(candidate):
                current = candidate
                granularity = max(2, granularity - 1)
                break
        else:  # no chunk could be removed at this granularity
            if granularity >= len(current):
                break
            granularity = min(len(current), granularity * 2)
    return current

Applied to an invoice that fails somewhere inside a 20-line order export:

items = [normal_1, normal_2, suspicious_discount_item, filler_3, filler_4]  # ... plus 15 more
minimal = minimize_failure(items, invoice_still_fails)
# -> [suspicious_discount_item]

Now MINIMIZE is executable. The reader can watch confounders fall away one chunk at a time instead of being told to “reduce the repro.” The real ddmin adds guarantees this version skips: 1-minimality — a failing case is 1-minimal iff removing any single element makes the failure disappear (pairwise-and-larger removals unchecked, so a 1-minimal case can still shrink by removing ≥2 at once) — and no handling of interference between jointly-necessary elements. The point here is only that each test is a question, not housekeeping.

Minimization is priced in trials. In their 2002 study the reductions cost 139 automated runs on a 500 MHz PC to shrink 95 user actions to three and an 896-line input to one line — worst case |c|²+3|c| tests, best case O(log n). Each still-fails verdict is itself an experiment, so where the oracle is noisy, repeat the cell until the verdict is stable (house floor: 3 trials; Ch21 sizes N from the forecasted effect; Ch58 gives the full noisy-oracle adaptation).

Two algorithms hide in one name. ddmin simplifies one failing input; dd bisects the difference between a passing and a failing configuration (Ch2’s bisection is dd over versions). ddmin’s correctness needs no monotonicity assumption — only its efficiency does — whereas Ch2’s bisection does need monotonicity, and a masking stage breaks it (Ch58). And minimization removes confounders; it does not vote on causes — a cause may be a conjunction of two elements, jointly necessary and neither singly removable, which is exactly ddmin’s interference case and Ch31’s pair-removal.

Research lineage: debugging as controlled experiment

The loop above sits on a long line of work in automated and empirical debugging, and that work did not just validate the loop — it shaped several sections of this chapter.

Zeller and Hildebrandt’s Delta Debugging formalized minimization as an algorithmic operation: repeatedly test smaller variants until the failure-inducing circumstances are minimal. Their 2002 study reduced a Mozilla failure from 95 user actions to three, and an 896-line HTML input to the single failure-inducing line. The lesson this chapter takes — that minimization is a sequence of experiments — drives the minimize_failure implementation above (Zeller & Hildebrandt, 2002). Zeller then carried the same intervention logic from inputs into program state, altering state differences between passing and failing runs to find which ones actually affect the outcome (Zeller, 2002). Cleve and Zeller sharpened this into cause transitions — the primitive introduced earlier in this chapter — and reported that transitions localized failure-inducing defects better than the comparison techniques in their evaluation (Cleve & Zeller, 2005).

Causal Testing made the counterfactual explicit and motivates the discount_override experiment below. On Defects4J, Johnson, Brun, and Meliou reported that the method was applicable to 71% of 139 determinable defects (of 330 examined across four Defects4J projects); for 77% of those it could supply root-cause-relevant information; and in a controlled experiment with 37 developers, participants identified the cause 86% of the time with Causal Testing versus 80% with standard testing tools. Those figures belong to that experimental setting — the paper’s 86/80, not the project page’s differing pair — not to debugging universally, but they support the narrower principle used throughout this book: carefully chosen counterfactual executions can improve diagnosis (Johnson, Brun & Meliou, 2020).

A field study by Siegmund and colleagues — eight professional developers across four companies, so suggestive rather than conclusive — observed the same hypothesis-then-verify pattern in real practice (Siegmund et al., 2014). That supports the loop as a teachable norm, not as a description of what experts do first: in high-validity niches (a familiar codebase with valid cues and fast feedback) experts plausibly recognize-first and simulate-first, and the loop’s function there is error-capture when recognition is wrong — recognition first where cues are valid; hypotheses first where they aren’t. And AutoSD ran the loop with an LLM proposing hypotheses and executable debugger experiments (Defects4J/Java via jdb plus BugsInPy/ARHE/Python via pdb; Codex/ChatGPT-era models) while the real program supplied observations; DONE-conditioned patches were 89% correctly fixed versus 82% without the completion signal, and in one ablation, replacing real debugger/code-execution results with model-predicted results flipped the signal — from +12.4pp more likely plausible with real execution to 11pp less likely plausible without it (per-run plausibility 73% → 63%) — a direct warning against letting the diagnosing model fabricate the evidence it is supposed to interpret (Kang et al., 2025).

A useful debugging hypothesis does not merely explain what you saw. It identifies an intervention whose possible outcomes would change what you believe.

That principle reappears for stochastic models, retrieval pipelines, and agent trajectories, where the intervention is a replay, an ablation, a context substitution, or a fork rather than a line of Python.

From scientific debugging to an AI diagnostician

Every loop in this chapter has the same shape, and it is the shape the rest of the book organizes its automation around — rigorous debugging can be organized as an experimental discipline resembling the scientific method (Zeller’s teachable fallback when ten minutes of intuition fail), not the ontology of what debugging is:

while not diagnosed:
    hypothesis  = diagnostician.propose(evidence)
    experiment  = diagnostician.design_test(hypothesis, evidence)
    observation = execute_real_test(experiment)   # runtime, not the model
    evidence.append(observation)
    diagnosed   = diagnostician.update(hypothesis, observation)

The AI proposes hypotheses and chooses tests. The runtime supplies observations. AutoSD built exactly this and found that the moment you let the model supply its own “observations” instead of executing them, its internal signal for “I’m done” starts pointing the wrong way. The division of labor is not a style choice:

model   : hypotheses, test selection
runtime : observations
Evidence never comes from the model's imagination.

Building the mental model: intent must be written down

The step practitioners skip most is writing down intent. “Intended execution” sounds abstract until you are staring at an LLM output or a discount boundary and realize nobody specified what correct means.

For deterministic code, intent is a table:

The table forces the discipline: the diagnosis effort goes to the discount line, not the tax line, even though the customer complained about the final amount. The reported symptom points to the end of the chain; the cause lives at the first break.

Sometimes no spec line covers the step you care about — nobody wrote what invoice_total should return for a subtotal of exactly €1,000, or whether a zero-item invoice is €0.00 or an error. You are still not stuck at UNKNOWN. A relation is an oracle that constrains two runs without stating either one’s exact output: a qualifying invoice and an otherwise-identical non-qualifying one must differ by exactly subtotal × 0.10; doubling every quantity must at least double the total; adding an item must never lower it. A violated relation convicts the arithmetic with no absolute answer in hand, and it is often easier to state than the missing spec line. This is the third oracle mode, after the intent table and the known-good diff below; Chapter 7 builds it into a technique (metamorphic testing) and Parts IV–VII lean on it where a per-output oracle never exists.

For AI systems later in the book, “intent” gets harder — a distribution, a citation requirement, a trajectory policy — but the move is identical: specify expected behavior before judging observed behavior, or every judgment is post-hoc storytelling.

Two runs, not one: diff then intervene

The intent table compares one failing run against a spec. When a second run is available — the same input through a known-correct version of the function — comparing the two is sharper, and it is the move the cause-effect-chain research is built on.

The diff has two differences. The interesting one is discount after branch: the value is right inside the branch and wrong immediately after it, which points at something between those two points rather than at the condition or the arithmetic. But the table still refuses to call that difference a cause until an intervention says so.

A diff tells you where two runs differ. An intervention tells you which difference matters.

Demonstration: localizing the invoice bug in five minutes

Here is Developer B’s actual session, compressed:

# repro.py — pinned reproduction
from billing import invoice_total

items = [{"price": 600, "qty": 1}, {"price": 600, "qty": 1}]
got = invoice_total(items)
print(f"got={got} expected=1296.0")
assert got == 1296.0, f"DIVERGENCE: got {got}"

OBSERVATION (run 1): got=1440.0 expected=1296.0. Failure reproduced.

She then instruments intermediates rather than guessing:

subtotal = sum(i["price"] * i["qty"] for i in items)
print("subtotal:", subtotal)  # 1200 — matches intent
print("condition:", subtotal >= 1000)  # True — so why discount 0?

Suppose the real file has a shadowing bug — the condition is correct but discount is reset two lines later:

    if subtotal >= discount_threshold:
        discount = subtotal * discount_rate
    discount = 0.0  # leftover from a merge — the actual defect

OBSERVATION: subtotal matches intent; discount is 0.0 immediately after the assignment block. HYPOTHESIS H1: condition never fires. HYPOTHESIS H2: discount assigned then overwritten. TEST: print discount inside the if block vs. after it. Prediction if H1: discount is 0.0 in both places. Prediction if H2: 120.0 inside, 0.0 after. OBSERVATION: 120.0 inside, 0.0 after. UPDATED BELIEF: H2 supported; H1 rejected. Candidate first divergence is the stray reassignment.

Note what happened: the fix location was determined by ordered evidence, not by which line “looked suspicious.” And the experiment was designed so each hypothesis predicted a different observable outcome. That is the standard for every lab in this book.

The experiment as a counterfactual

The probe above compared discount inside and outside the branch. The stronger form, from the Causal Testing line of work, is an explicit counterfactual: hold every other input and state constant, set the suspect value to what it should be, and see whether the failure moves.

def invoice_total(items, *, discount_override=None):
    subtotal = sum(i["price"] * i["qty"] for i in items)
    discount = subtotal * 0.10 if subtotal >= 1000 else 0.0
    discount = 0.0                       # the defect
    if discount_override is not None:    # counterfactual hook
        discount = discount_override
    total = subtotal - discount
    return round(total * 1.20, 2)

observed        invoice_total(items)                          -> 1440.0
counterfactual  invoice_total(items, discount_override=120.0) -> 1296.0

The failure moves from 1440 to the expected 1296 when — and only when — discount is forced to its intended value. That is stronger evidence that discount is on the causal path than any amount of reading the code. It is still not proof: the override could be masking a second defect downstream. Confidence, not certainty.

Two rules select the next experiment without any new machinery. Discriminate first: prefer the test whose possible outcomes eliminate the most live hypotheses — the inside-vs-after probe above eliminates H1 XOR H2 in one run, which is why it is the example. Price information in trials: an experiment’s cost is its trial count × oracle cost; prefer the cheapest discriminator. Ch47/48 formalize these; here they are habits.

An epistemic state machine for hypotheses

“H2 confirmed” is too binary to carry through the harder chapters. Track each hypothesis on a ladder:

OBSERVED            a value differs from intent
   ↓
HYPOTHESIZED        a named mechanism that would explain it
   ↓
SUPPORTED           evidence is consistent with it; alternatives not yet excluded
   ↓
CAUSALLY SUPPORTED  an intervention changed the outcome as predicted
   ↓
CONFIRMED           holds on replication AND on the reverse intervention
                    (undoing the fix restores the failure)

Side states: REFUTED (an intervention falsified it — reject confidently); INCONCLUSIVE (no clean intervention is possible — transient, with the missing envelope element logged as the reason); PROVISIONAL (capped: multiple contributors, or not safely reversible). The asymmetry is deliberate and returns in Ch41: rejection can be confident, confirmation stays cautious — forward intervention establishes sufficiency-in-context, reverse establishes necessity-in-context, and neither establishes universality. And note the rename: the top rung is CONFIRMED, not REPRODUCED — “reproduced” already means the failure itself reruns (Ch45), while CONFIRMED is a statement about the hypothesis. Irreversible interventions (production, destructive tests, unreproducible states) can never reach CONFIRMED — they cap at CAUSALLY SUPPORTED or PROVISIONAL. CONFIRMED’s replication leg is ≥1 clean re-run here, and an N-trial paired-seed series where the oracle is noisy (Ch21). A cause may be a conjunction of jointly-necessary elements (record the set, not a single line); an inseparable conjunction caps at PROVISIONAL.

A hypothesis record, which later chapters formalize:

hypothesis:
  id: H2
  claim: discount is assigned then overwritten
  status: SUPPORTED            # -> CAUSALLY SUPPORTED after the override test
  evidence:
    - discount=120.0 inside branch
    - discount=0.0 after branch
  next_test: force discount via counterfactual override; expect 1296.0
root_cause:
  status: UNCONFIRMED

The ladder is what keeps “looks wrong” and “proven cause” from being written down with the same confidence. In one picture — the shape to carry through every later chapter:

    flowchart TD
    D[Observed difference] --> R{"Relevant to the failure?"}
    R -->|no| X["Discard — a difference, not a lead"]
    R -->|yes| I{"Intervention changes the outcome as predicted?"}
    I -->|no| W["Hypothesis weakened / REFUTED"]
    I -->|yes| C[Causally supported transition]
    C --> V[Reversal / repeat / alternative test]
    V --> K["CONFIRMED — within the stated envelope"]
    C -.->|"cannot intervene, reproduce, order, or measure"| P["PROVISIONAL (multi-cause / irreversible) or INCONCLUSIVE (no clean intervention)"]
  

Two vocabularies, two jobs — do not conflate them. The row-labels from the book-wide rules (OBSERVATION, HYPOTHESIS, INFERENCE, MEASUREMENT, EXPERIMENTAL RESULT, FORECAST, OPINION, UNKNOWN; Ch3) annotate a line of evidence. The ladder above tracks a hypothesis. A hypothesis at SUPPORTED is built from rows labelled OBSERVATION and MEASUREMENT; a single claim checked against a single context gets the narrower evidence-chain verdicts SUPPORTED / UNSUPPORTED / CONTRADICTED / OUT-OF-CONTEXT (Ch34) — an instrument for one claim, not a competing general vocabulary.

Lab 1: reproduce → minimize → intervene

Run the whole progression the research describes, on the invoice_total function with the stray-reassignment defect inserted. Pin your Python version and the file hash in your notes.

Level 1 — reproduce. Write repro.py that fails deterministically on a two-item €1,200 invoice. Run it 5× and log the trial counts (5/5). Success: the same wrong number on every run.

Level 2 — minimize. Start from a 20-item order export that fails. Use minimize_failure (above) with your reproduction as still_fails. Log per-cell trial counts plus the total candidate lists tested. Then swap in one deliberately-flaky still_fails variant (seeded ~20% pass) and watch 1-trial minimization isolate the wrong element — the noisy-oracle lesson in miniature. Success: a one- or two-item list that still fails, plus a note of how many candidate lists were tested to get there.

Level 3 — causal intervention. Two hypotheses are live: (H1) wrong comparison operator, (H2) discount overwritten downstream. Design one intervention whose predicted outcome differs under H1 and H2 — the counterfactual discount_override is one option, an inside-vs-after probe is another. Then run the interaction counterexample: a two-line config where neither line alone fails — run the single-variable tests (both refuted; naive single-cause reading says INCONCLUSIVE), run pair-removal (failure vanishes), and record the CONJUNCTIVE cause on the ladder.

• Independent variable: exactly one change.
• Controlled variables: same input, same code version except the probe, same environment.
• Write predictions before running:
  • Prediction if H1:
  • Prediction if H2:
• Run, record OBSERVATION, move the hypothesis along the ladder.
• File the envelope line with every lab note: input hash, code version, trial counts, oracle stability — plus the cap (CONFIRMED / PROVISIONAL / INCONCLUSIVE) where applicable.

Level 4 — scope boundary + INCONCLUSIVE case. Take a failure with no clean intervention (e.g. the suspect value entangled with two others through a shared computation, so the override produces an impossible state). Required output: a logged INCONCLUSIVE naming the missing envelope element (“state not independently settable”), not a guess. This normalizes the honest outcome on day one.

Level 5 — reversal + regression. Delete the stray line, re-run repro + boundaries 999/1000/1001 + suite, re-add the line and confirm the failure returns (the reverse leg — mandatory for CONFIRMED), then file the four-part done record + regression test.

What counts as success. Not “fixed the bug.” A lab note of this form: “Prior H1 0.5 / H2 0.5. Intervention: override discount=120. Observation: 1440 → 1296. Posterior H1 ~0 / H2 ~1 (CAUSALLY SUPPORTED). Next: delete line N, rerun repro + boundaries 999 / 1000 / 1001.”

PROPOSED EXPERIMENT: Lab 1 is specified here for the reader to run. No measured results are claimed in this chapter. Do not report outcomes you did not observe.

Companion tool: Debugging Definition Checklist

Every chapter in this book ships with a companion tool — an executable form of the chapter’s idea. Chapter 1’s tool is a checklist, because at this stage the failure mode is skipping steps, not lacking tooling.

What it accepts: a symptom description, a written intent statement, a reproduction script path. What it performs: a gate check — you may not advance to hypotheses until observation, reproduction, and minimization are recorded. What it can establish: whether your diagnosis process is complete enough to trust. What it cannot establish: the cause itself. A checklist prevents self-deception; it does not find bugs. How its output changes your next action: any unchecked box is your next action. No hypothesis work until reproduction is pinned.

[ ] Intent written as measurable expectation (value + tolerance)?
[ ] Observed value captured with version + input hash?
[ ] Reproduction script reruns the failure on demand?
[ ] Minimized to smallest failing input?
[ ] First divergence identified (not just final symptom)?
[ ] ≥2 competing hypotheses recorded?
[ ] Intervention + prior prediction written before running?
[ ] Hypothesis placed on the OBSERVED→CONFIRMED ladder (reverse intervention for CONFIRMED)?
[ ] Verification: repro + edge cases + suite?
[ ] Prevention artifact created (test / assertion / note)?

OPINION: teams that enforce this checklist close fewer tickets per day for one week, then permanently fewer regressions. The slowdown is real and worth it.

When the experiment cannot isolate a cause

Causal Testing was applicable to 71% of the real-world defects its authors studied — not all of them. Some failures resist clean intervention:

the suspect value cannot be set independently of others
setting it produces an impossible program state
several variables move together and cannot be separated
the failure will not reproduce reliably
measuring the value changes the execution

The correct output in those cases is a recorded INCONCLUSIVE — the hypothesis is neither confirmed nor refuted, and the reason is logged — not “guess harder” and not a fix shipped on a hunch. The category is impossible or inseparable states: some interventions correspond to no real execution. Later chapters hit this constantly with stochastic models and agents; naming it now makes it a normal outcome rather than a failure of nerve.

Failure modes and misleading interpretations

• Single-run inference. “It passed once after my edit, so it’s fixed.” One run distinguishes nothing — flakiness, caching, and coincident inputs all produce single passes. Verification requires repetition plus edge cases.
• Post-hoc storytelling. “The tax line looked wrong, and changing it helped, so that was the bug.” A story constructed after the edit is not evidence. Predictions must precede observations.
• Changing multiple variables at once. Editing the comparison and the rounding and upgrading a dependency in one go means the outcome teaches nothing. One variable per experiment, or the result is uninterpretable.
• Confusing a diff with a diagnosis. Two runs differ in many places; most differences are irrelevant. A difference is a lead until an intervention promotes it. Ranked locations alone do not reliably help — developers do not traverse rankings in order and understanding needs dependency navigation plus re-execution (Parnin & Orso, ISSTA 2011); candidate hypotheses beat candidate locations (Ch6).
• Treating an explanation as an observation. Later chapters extend this to models: an LLM’s self-report (“I applied the discount”) is an explanation, not a trace. Chapter 1’s version: a code comment claiming intent is not intent. The spec and the run are evidence; everything else is hypothesis.

What counts as “done” for a diagnosis

A Chapter-1-complete diagnosis has four parts:

1. Divergence statement: “First divergence at discount assignment: intended 120, observed 0.0, on input X at code version Y.”
2. Causal test: “Forcing discount to 120 via override changes output from 1440.0 to 1296.0 as predicted; removing the stray reassignment does the same; re-adding it restores the failure.”
3. Scope statement: “Boundary inputs 999 / 1000 / 1001 behave as specified after fix; full suite passes.”
4. Prevention artifact: a regression test asserting the discounted and non-discounted boundaries.

Without all four, you have a fix candidate, not a diagnosis.

One softening, carried from research the later chapters develop (Ch49, Ch59): a “root cause” in this book is an evidence-backed diagnostic construction — the earliest divergence that survives a causal test, scoped to the examined input, version, and trials — not an omniscient reconstruction of every contributing factor. This chapter diagnoses a local defect in a deterministic program; incidents with people, organizations, and incentives need Part X’s machinery (Ch49’s hindsight-bias critique, Ch59’s separate contributing-factors list) — the discipline transfers, the noun doesn’t. In deterministic single-defect cases the construction is tight. In socio-technical incidents, distributed systems, agent failures, and multi-factor breakdowns, contributing factors stay a separate list and the honest cap is often PROVISIONAL. The discipline does not weaken; the noun gets honest.

Three things this chapter does not claim — and the book never will. First, one successful run after an edit does not prove universal causality; it proves the intervention changed this reproduction, nothing more. Second, no model-generated explanation may substitute for preserved evidence; a fluent account of “why” is a hypothesis until an intervention test confirms it. Third, none of this eliminates human verification in high-impact decisions: the checklist and the regression test inform judgment, they do not replace sign-off where money, safety, or production traffic is at stake.

References

• Andreas Zeller and Ralf Hildebrandt. Simplifying and Isolating Failure-Inducing Input. IEEE Transactions on Software Engineering 28(2), 2002, pp. 183–200. https://doi.org/10.1109/32.988498
• Andreas Zeller. Isolating Cause-Effect Chains from Computer Programs. Proceedings of the Joint ESEC/FSE Conference, 2002, pp. 1–10. https://doi.org/10.1145/587051.587053
• Holger Cleve and Andreas Zeller. Locating Causes of Program Failures. Proceedings of the 27th International Conference on Software Engineering (ICSE), 2005, pp. 342–351. https://doi.org/10.1145/1062455.1062522
• Benjamin Siegmund, Michael Perscheid, Marcel Taeumel, and Robert Hirschfeld. Studying the Advancement in Debugging Practice of Professional Software Developers. IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), 2014, pp. 269–274. https://doi.org/10.1109/ISSREW.2014.36
• Brittany Johnson, Yuriy Brun, and Alexandra Meliou. Causal Testing: Understanding Defects’ Root Causes. Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (ICSE), 2020. https://doi.org/10.1145/3377811.3380377
• Sungmin Kang, Bei Chen, Shin Yoo, and Jian-Guang Lou. Explainable Automated Debugging via Large Language Model-Driven Scientific Debugging. Empirical Software Engineering 30, 45 (2025). https://doi.org/10.1007/s10664-024-10594-x
• Chris Parnin and Alessandro Orso. Are Automated Debugging Techniques Actually Helping Programmers? Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), 2011. https://doi.org/10.1145/2001420.2001445
• Michelene Chi, Paul Feltovich, and Robert Glaser. Categorization and Representation of Physics Problems by Experts and Novices. Cognitive Science 5(2), 1981, pp. 121–152. https://doi.org/10.1207/s15516709cog0502_2

Debugging Checklist

• Did I write intent before judging observed behavior?
• Can I retrigger the failure with one command?
• Did I minimize to the smallest failing input?
• Did I identify the first divergence, not just the symptom?
• Are observation, hypothesis, and inference in separate notes?
• Did I run a counterfactual intervention before calling anything a cause?
• Did I verify beyond the single failing input?
• Did I leave a prevention artifact behind?

What This Chapter Established

• Debugging is defined as constructing the smallest evidence-backed causal account that predicts the failure and its reversal — inside a pinned input × version × oracle envelope, by finding the earliest transition surviving forward and reverse intervention; outside the envelope, by difference → relevance → support capped at PROVISIONAL or INCONCLUSIVE — distinct from making a symptom disappear.
• “Earliest divergence” resolves into three levels: first difference, first relevant difference, first causal divergence; only the third earns the word cause. The cause transition — where the cause set changes membership (one variable ceases, another begins) — is distinct from both the defect location and the failure location.
• Reproduction quality bounds diagnosis quality; observation must precede explanation; minimization is a sequence of experiments (minimize_failure); the experiment is strongest as an explicit counterfactual.
• Hypotheses move along an OBSERVED → HYPOTHESIZED → SUPPORTED → CAUSALLY SUPPORTED → CONFIRMED ladder (CONFIRMED requires replication plus the reverse intervention); refutation is confident, confirmation cautious; irreversible interventions cap at CAUSALLY SUPPORTED or PROVISIONAL; when intervention is impossible the honest output is INCONCLUSIVE, and multi-cause or irreversible cases cap at PROVISIONAL.
• The five debugging objects (values → state → distributions → evidence → trajectories) order the book’s progression; this chapter operated on values only. They are PROPOSED as deep-structure categories, on the Chi analogy: experts sort failures by what kind of thing is failing rather than by surface symptom (Chi, Feltovich & Glaser, 1981 — undergrad physics sorting; no debugging-outcome study in this book’s set, full treatment in Ch60), so learning the objects is learning to route like an expert.
• Prior debugging research (delta debugging, cause-effect chains, cause transitions, Causal Testing, field studies of developers, AutoSD) supports these strategies; the invoice example remains a constructed illustration and Lab 1 remains proposed, not executed.

Next

Chapter 1 defined what to find — the earliest causal divergence. It did not teach how to find it efficiently in a longer execution. Chapter 2, “The First Divergence,” turns the definition into a localization procedure: checkpointing, bisection, and walking ordered intermediates so the cause is isolated before any hypothesis is indulged.