Part I — Debugging From First Principles

The puzzle: the report is wrong, but where?

Chapter 1 defined debugging as constructing the smallest evidence-backed causal account that predicts the failure and its reversal — pursued, inside a pinned envelope, by finding the earliest transition surviving forward and reverse intervention — and split “earliest divergence” into three levels: first difference, first relevant difference, first causal divergence. This chapter makes the localization part executable — how you get to a candidate first divergence efficiently, before the hypothesis cycle promotes it to a cause.

New scenario. A nightly revenue report is off by €212.40 on 3,842 orders. The report pipeline has five stages:

ingest → clean → aggregate → discount → render

The obvious move is to stare at render — that is where the wrong number appears. A developer adds a rounding fix in render, the total shifts by a few euros, and the ticket is closed as “floating-point noise.” Next night, the report is wrong again by a different amount.

OBSERVATION: the final total is wrong. HYPOTHESIS (unearned): the last stage caused it. INFERENCE: none yet — no intermediate evidence has been examined.

The symptom points at the end of the chain. The cause almost never lives there. This chapter teaches the localization procedure that Chapter 1 assumed: walk ordered intermediates, mark the first mismatch, and stop confirming anything downstream of it — theorizing may start anywhere; confirmation may not.

Why the obvious explanation fails

Three runs of the same pipeline can produce the same wrong total for three different reasons:

• A. clean drops 14 refunded orders it should keep as negative lines → subtotal too high from step 2 onward.
• B. aggregate double-counts orders with split shipments → quantity inflation from step 3 onward.
• C. discount applies a stale promo code → totals diverge only at step 4.

Same surface symptom (wrong final number), three different first divergences, three different fixes. Any diagnosis that starts from the final number and reasons backward without checkpoints cannot separate A, B, and C. That is the whole point: final-output reasoning underdetermines the cause in any multi-stage execution.

What separates them is not cleverness but order. If you capture intended vs. observed at each stage boundary, exactly one stage is the first to mismatch. Everything after it is effect.

The method: checkpoint the chain, bisect toward the break

A divergence checkpoint is a recorded comparison at a stage boundary: intended value, observed value, verdict. For deterministic code, “intended” comes from the spec, a prior known-good run, or a hand-computed fixture. For later chapters (distributions, evidence, trajectories), intent gets probabilistic — but the checkpoint habit is identical.

When a boundary has no intended number — no spec line says what the post-clean revenue should be — it usually still has an intended relation (Chapter 1’s third oracle mode). Order count must not rise across clean; row count out of aggregate cannot exceed row count in; total revenue must be non-negative; the sign of each line must be preserved. A checkpoint that records “relation held / violated” localizes exactly as well as one that records a number, and relations are available at boundaries where a golden value never was.

The procedure:

1. Enumerate the stages in execution order. Not in order of suspicion — in order of execution. Write them down: ingest(n) → clean(n) → aggregate(n) → discount(n) → render(n).
2. Fix the reproduction. Same input snapshot, same code version, same seed/config. If the input moves between runs, checkpoints are incomparable.
3. Capture each boundary value once, without editing anything. Logging probes only; no fixes yet.
4. Compare against intent in order, and stop at the first mismatch. That stage owns the investigation. Stages after it are quarantined — interesting, but not causal.
5. If the chain is long, bisect. Check the middle boundary first. If it matches, the break is downstream; if it mismatches, it is upstream or at the midpoint. Recurse. Bisection finds the first divergence in O(log n) probes instead of O(n) — this is Zeller’s dd over stages rather than inputs (Ch1’s ddmin simplifies one input; Ch58 adapts both to noisy oracles, where each still-fails verdict is repeated until stable — house floor 3 trials).

    flowchart LR
    I[ingest: ok] --> C[clean: ok?] --> A[aggregate: ok?] --> D[discount: ok?] --> R[render: WRONG]
    C -.->|first mismatch here| FIX[all downstream is effect]
  

The precondition bisection needs: monotonicity. Binary search over a chain — like git bisect over a commit history, which is the same algorithm applied to versions instead of stages — only works if “broken” stays broken once it starts. If a downstream stage can partially cancel an upstream divergence, the verdict sequence stops being match, match, MISMATCH, mismatch, mismatch and starts flickering. Then the middle probe lies. Before trusting bisection, check that each boundary’s error is at least as large as the previous one; a boundary where the error shrinks is a masking stage and must be probed linearly, not bisected.

Concretely, compare two verdict columns:

bisection-safe (monotone):   ok    ok    +€212   +€212   +€212
masked (flickering):         ok   +€212  +€6     +€44    +€212

In the masked column, a first probe at the middle boundary reads +€6 — nearly right — and sends the search downstream, past the real break at stage 2. The error was cancelled at stage 3 (a clamp, a re-round, a re-aggregation that hides the upstream damage) and re-emerged later. In the revenue table the euro error grows monotonically after clean, so bisection is safe there; whenever the column is not monotone, walk it linearly.

In the revenue-report case, the checkpoint table looked like this (simplified to order counts and revenue):

The investigation now belongs to clean, not render. The render-rounding “fix” is exposed as symptom-chasing: it moved the final number without touching the first break.

Demonstration: the fourteen missing orders

With the divergence localized to clean, the code gets a focused reading — two hypotheses, not twenty:

def clean(orders):
    out = []
    for o in orders:
        if o["status"] == "refunded":
            continue          # suspect line: drops refunds entirely
        out.append(normalize(o))
    return out

Spec says refunded orders must be kept as negative-amount lines so aggregates net correctly. The continue deletes them.

H1: clean wrongly drops refunded orders (first divergence at clean). H2: aggregate double-counts split shipments (first divergence at aggregate). TEST: rerun pipeline on a 10-order fixture containing 2 refunds and 1 split shipment, with per-stage counts logged. Prediction if H1: post-clean count is 8, post-aggregate revenue is low by exactly the refund amounts. Prediction if H2: post-clean count is 10, post-aggregate quantity exceeds 10. OBSERVATION (illustrative, constructed): post-clean count 8; revenue shortfall equals the two refunds. UPDATED BELIEF: H1 confirmed; H2 rejected for this incident.

Two notes on honesty. First, the numbers above are a constructed illustration of the method, not a measured production incident — this chapter claims no empirical result. Second, confirming H1 for this incident does not preclude H2 existing as a second latent defect; bisection finds the first divergence, then you re-checkpoint after fixing it and look for the next.

The fix is one-line in spirit (keep refunds as negatives), but the diagnosis artifact is what matters:

def test_clean_keeps_refunds_as_negatives():
    orders = [{"id": 1, "status": "refunded", "amount": 100.0}]
    assert clean(orders) == [{"id": 1, "status": "refunded", "amount": -100.0}]

Within the stage: the backward slice

“Confine your reading to the diverging stage” is still vague if the stage is 400 lines. The precise version of that instruction is the backward dynamic slice of the wrong value: the set of statements that actually influenced this output on this input, following data and control dependencies backward from the divergence point. Everything outside the slice provably did not contribute and can be ignored while reading.

For a data pipeline the slice is a provenance query — which input rows fed the wrong aggregate?

def contributing_orders(orders, predicate):
    """Backward slice, data-pipeline flavour: rows that reached the wrong bucket."""
    return [o for o in orders if predicate(o)]

# the €212.40 shortfall traces to exactly these:
guilty = contributing_orders(raw_orders, lambda o: o["status"] == "refunded")
assert len(guilty) == 14
assert round(sum(o["amount"] for o in guilty), 2) == 212.40

The shortfall now has a named set of inputs behind it, not a stage. That is the difference between “the bug is in clean” and “the bug is clean discarding these 14 rows whose amounts sum to the exact error.”

Research lineage: localization is not comprehension

Two research traditions bear on this chapter, and they pull in opposite directions.

Statistical fault localization ranks program elements by how strongly their execution correlates with failure across a test suite. Jones and Harrold’s Tarantula and the Ochiai coefficient adapted from molecular biology by Abreu and colleagues are the canonical scores — Ochiai scores a statement as a_ef / sqrt((a_ef + a_nf) * (a_ef + a_ep)) from failing/passing coverage counts, and beat Tarantula and Jaccard on the Siemens-suite evaluation in that study; Wong and colleagues later surveyed the whole spectrum- and mutation-based family (Jones & Harrold, 2005; Abreu, Zoeteweij & van Gemund, 2006; Wong et al., 2016). This is the alternative to ordered checkpointing: instead of one trusted trace compared against intent, it uses many traces and a correlation statistic. It scales to code with no per-stage spec, at the cost of producing a ranking rather than a first divergence. Note the warrant asymmetry: checkpointing is deductive given correct per-boundary intent (the first mismatch in a total execution order is the first break by construction); ranking is heuristic and needs benchmarks, which is where its troubles start.

Program slicing is the older idea behind the previous section. Weiser introduced static slicing as a debugging aid; Korel and Laski made it dynamic — restricted to one execution — which shrinks the slice to only the statements that ran and actually mattered (Weiser, 1984; Korel & Laski, 1988). For database pipelines the same “work backward from a bad output” instinct became Scorpion, which explains an aggregate outlier by finding a predicate over input tuples that, removed, makes the outlier disappear — a direct analogue of contributing_orders above (Wu & Madden, 2013).

The disconfirming result is the important one. Parnin and Orso ran two user studies on whether ranked suspiciousness lists actually help developers. They found that developers do not scan the list top-to-bottom, that a fault ranked in the “top 10%” is useless if that still means inspecting dozens of statements, and that being shown the faulty line does not mean the developer understands the fault — the “you found it, so you’re done” assumption fails in practice (Parnin & Orso, 2011). The metrics side tells the same story: Pearson and colleagues replicated ten fault-localization claims and found every one refuted or insignificant on real faults — artificial (mutant) faults do not predict which technique wins on real faults, differences between spectrum-based techniques on real faults are negligible, and better EXAM scores (rank over program size) did not always mean faster human debugging; their novel hybrids win specifically at reporting defects in the top 5–10 slots (Pearson et al., 2017). That refutation is scoped to ranking techniques — the authors explicitly exempt slice-based techniques, which protects the backward-slice half of this chapter. The operational rule both studies imply: consume a ranking as a ~10-item triage list, never a diagnosis. This is exactly why the companion tool’s contract says first divergence is a location, not a cause, and why localization in this book is always followed by the hypothesis–intervention cycle rather than treated as the answer. Where no per-stage intent exists at all, ranking is the fallback move — capped at SUPPORTED-at-best on Chapter 1’s ladder, never CAUSALLY SUPPORTED.

From ordered checkpoints to automated localization

Recent work automates the localization step with an LLM. AutoFL gives a model function-call tools to navigate a repository and asks it to both name a fault location and explain it; on 798 real Java and Python bugs the authors reported method-level acc@1 improvements up to 233% over prior techniques, and — echoing Parnin and Orso — interviewed developers preferred a few high-quality explanations to a long ranked list (Kang, An & Yoo, 2024).

The lesson for this book’s architecture: the checkpoint capture is mechanizable and the explanation is what a human actually wants, but the model must still get its boundary values from a real run. An LLM that guesses the post-clean count instead of executing the stage is back to final-output reasoning with extra steps.

Lab 2: bisect a five-stage pipeline

Setup. Take any deterministic multi-step script you own (ETL, report, invoice batch). If you have none, construct one with five pure functions and inject one defect in stage 2 or 3. Record the code hash.

Task. Without reading the injected defect, localize it by checkpoints alone.

• Independent variable: the boundary you probe (start with the middle stage).
• Controlled variables: fixed input snapshot, fixed code version, probe-only logging (no edits during localization).
• Competing hypotheses before probing: H1 “break at or before midpoint” vs. H2 “break after midpoint.”
• Write both predictions before running. Run. Record OBSERVATION. Recurse to the first mismatch.
• Before recursing, check monotonicity: does the error size grow (or hold) at each successive boundary? If it shrinks anywhere, that stage masks — probe linearly through the masked region instead of bisecting.
• Then — and only then — take the backward slice of the wrong value at that stage (or the provenance of the wrong rows) and confine your reading to it before proposing the fix.

Success criterion. A note of the form: “Checked boundary 3/5 first: mismatch → narrowed to stages 1–3. Checked 2/5: match → first divergence at stage 3. Code read confined to stage 3; defect found at line N.” If you read the whole file first, you practiced reading, not localization.

PROPOSED vs. EXECUTED: this lab is proposed for the reader. No measurements from the author’s runs are reported here because none are needed — the claim is methodological, and the evidence it requires is the reader’s own checkpoint table.

Companion tool: First Divergence Finder

What it accepts: an ordered stage list, a pinned input reference, and per-boundary intended values (spec numbers or a known-good snapshot). What it performs: it runs the reproduction, captures each boundary value, and reports the first mismatch in execution order — optionally suggesting the next bisection probe for long chains. What it can establish: where observation first departs from intent under this input and version. What it cannot establish: why. “First divergence at clean” is a location, not a cause. Cause requires the hypothesis–intervention cycle of Chapter 1. How its output changes your next action: confine all code reading and hypothesis generation to the diverging stage. Downstream stages are off-limits until the first break is fixed and re-checkpointed.

A minimal contract, in pseudocode rather than a false implementation claim:

finder --stages ingest,clean,aggregate,discount,render \
       --input snapshot-2026-09-07.csv \
       --expect known-good.json \
       --report first-divergence.json
# report: { "first_mismatch": "clean", "expected": 3842, "observed": 3828 }

Where the tool does not yet exist in the reader’s stack, the checkpoint table on paper is the tool. The discipline precedes the automation.

Failure modes

• Symptom-anchoring. Debugging the stage where the complaint surfaced instead of the stage where the chain first broke. The customer always reports the end; the cause is upstream.
• Skipping boundaries. “I read all five stages and stage 4 looked worst.” Reading is not checkpointing. Without recorded intended-vs-observed per boundary, suspicion is not localization.
• Fixing downstream. Patching render rounding when clean drops rows. The number moves; the defect remains. Re-checkpointing after every fix catches this.
• Single-fixture generalization. “My 10-order fixture localized it, so all failures are this defect.” One fixture separates the hypotheses on the table; it does not enumerate all defects. Re-run the full suite plus the original failing snapshot.

Three limits, carried forward from the contract. Localizing the first divergence on one input does not prove universal causality — it proves where this reproduction first breaks. A checkpoint table is preserved evidence, not a substitute for model-generated explanations of why the break occurred. And in high-impact settings, a localized cause still requires human verification before the fix ships: bisection directs judgment, it does not replace it.

References

• Mark Weiser. Program Slicing. IEEE Transactions on Software Engineering SE-10(4), 1984, pp. 352–357. https://doi.org/10.1109/TSE.1984.5010248
• Bogdan Korel and Janusz Laski. Dynamic Program Slicing. Information Processing Letters 29(3), 1988, pp. 155–163. https://doi.org/10.1016/0020-0190(88)90054-3
• James A. Jones and Mary Jean Harrold. Empirical Evaluation of the Tarantula Automatic Fault-Localization Technique. Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2005, pp. 273–282. https://doi.org/10.1145/1101908.1101949
• Rui Abreu, Peter Zoeteweij, and Arjan J. C. van Gemund. An Evaluation of Similarity Coefficients for Software Fault Localization. Proceedings of the 12th Pacific Rim International Symposium on Dependable Computing (PRDC), 2006, pp. 39–46. https://doi.org/10.1109/PRDC.2006.18
• W. Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. A Survey on Software Fault Localization. IEEE Transactions on Software Engineering 42(8), 2016, pp. 707–740. https://doi.org/10.1109/TSE.2016.2521368
• Spencer Pearson, José Campos, René Just, Gordon Fraser, Rui Abreu, Michael D. Ernst, Deric Pang, and Benjamin Keller. Evaluating and Improving Fault Localization. Proceedings of the 39th International Conference on Software Engineering (ICSE), 2017, pp. 609–620. https://doi.org/10.1109/ICSE.2017.62
• Chris Parnin and Alessandro Orso. Are Automated Debugging Techniques Actually Helping Programmers? Proceedings of the 2011 International Symposium on Software Testing and Analysis (ISSTA), 2011, pp. 199–209. https://doi.org/10.1145/2001420.2001445
• Eugene Wu and Samuel Madden. Scorpion: Explaining Away Outliers in Aggregate Queries. Proceedings of the VLDB Endowment 6(8), 2013, pp. 553–564. https://doi.org/10.14778/2536354.2536356
• Sungmin Kang, Gabin An, and Shin Yoo. A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization. Proceedings of the ACM on Software Engineering 1 (FSE), 2024, article 79. https://doi.org/10.1145/3660771

Debugging Checklist

• Stages listed in execution order (not suspicion order)?
• Reproduction pinned (input + version) before probing?
• Every boundary captured with probe-only logging?
• First mismatch identified by ordered comparison?
• Monotonicity checked before bisecting (no masking stage)?
• Bisection used for long chains (middle probe first)?
• Code reading confined to the backward slice / provenance of the wrong value?
• Competing hypotheses with distinct predictions recorded?
• Localization treated as a location, not an understanding, before the hypothesis cycle?
• After fix: re-checkpointed end-to-end for a second break?

What This Chapter Established

• Final-output reasoning cannot separate causes that share a symptom; ordered checkpoints can.
• The first divergence localizes investigation to one stage; downstream mismatches are effects until proven otherwise.
• Bisection makes localization logarithmic in chain length — but only when brokenness is monotone; a masking stage forces a linear scan.
• Within a stage, the backward dynamic slice (or, for data pipelines, the provenance of the wrong value) is the principled bound on what to read.
• Research supports both sides of a tension: statistical fault localization (Tarantula, Ochiai) scales to specless code but only ranks — and the replication literature (Pearson et al.: mutant-evaluated rankings do not transfer to real faults; inter-SBFL differences negligible; EXAM scores do not track human speed) bounds exactly what rankings can claim; and user studies (Parnin & Orso) show a ranked location is not a diagnosis. Localization always precedes, never replaces, the hypothesis–intervention cycle.
• Lab 2 and all numeric tables here are constructed illustrations of method, not measured production results.

Next

Checkpoints assume the recorded values are trustworthy evidence. But what happens when the “evidence” is a fluent explanation — a comment, a log message, or a model telling you what it did? Chapter 3, “Evidence Before Explanation,” introduces the hygiene that keeps explanations from contaminating observations — the prerequisite for debugging anything a model generates.