Part X — The Debugging AI Playbook

One hour, one defect, pins available

Chapter 57 ended with a routed handoff: the flagged refusal triaged, pins UNKNOWN, no verdict, hour queue. Now the practitioner sits down with sixty protected minutes, log access, pinned environment, and the handoff record. The failure is concrete: the research agent’s answers degraded after Tuesday’s index refresh — some queries cite stale policy, others refuse outright. One hour is the book’s standard depth: enough to isolate, not enough to certify.

OBSERVATION: post-refresh traces show stale citations and refusals on overlapping queries; the handoff freeze holds input/output text but pins were UNKNOWN; index snapshot changed Tuesday (logged), model identifier unchanged (logged). HYPOTHESIS H1 (retrieval regression): the refreshed index returns stale or missing chunks that the fixed prompt cannot survive. H2 (assembly interaction): the index is fine; truncation or template assembly drops the fresh chunks before the model. H3 (prompt fragility): retrieval and assembly are adequate; the prompt has no stated policy for conflicting or absent citations, so behavior scatters. INFERENCE: none yet — H1/H2/H3 predict different stage-isolation signatures and separate within the hour by single-variable swaps.

This chapter’s question: what does the standard-depth procedure isolate in one hour — and what does it honestly hand upward?

Why “an hour means a root cause” fails first

The obvious move — expecting a certified cause with a shipped fix in sixty minutes — fails because isolation is not certification. Five overreaches hide behind hourly verdicts:

1. Bisected-but-unrepeated. The stage isolated on single runs, trials skipped “for time.” Nondeterminism unconvinced; the isolation is provisional.
2. Two-variable swaps. Index and prompt changed together to “save a round.” Attribution destroyed at the moment of success.
3. Fixture-free fixes. A repair shipped on the failing case alone, no suite run. The hour fixes the instance and schedules the recurrence.
4. Calibration theater. A guardrail threshold set from the hour’s three samples. Thresholds need calibration sets (Ch54); three samples are anecdotes.
5. Postmortem skipping. “Small incident, no writeup.” The hour’s evidence dies in the investigator’s head; Chapter 53’s conveyor starves.

OPINION: an hour buys an isolated divergence with a queued artifact. Call it a root cause and the next incident collects the difference.

The mental model: the one-hour investigation as standard isolation — consume the handoff, bisect input vs. environment, minimize to the divergence, verify across trials, and queue the artifact. KEPT at this budget: full record assembly (Ch52 fields), input-vs-environment bisection (Ch30), single-variable minimization, ≥3 trials per cell, a repair candidate validated on a small suite, and a queued regression test. SKIPPED honestly: full calibration sets, cost/latency joint optimization, multi-edge guardrail rollout, postmortem publication, and cross-incident generality claims.

The hour’s two core moves are Zeller’s two delta-debugging algorithms (Zeller & Hildebrandt, Simplifying and Isolating Failure-Inducing Input, IEEE TSE 2002). Quarter 2 is dd — isolate the minimal difference between a passing configuration (pre-refresh index + old prompt) and a failing one, one element of the change set at a time; git bisect is the same algorithm run over commit history. Quarter 3 is ddmin — remove input until every remaining piece is load-bearing. Both algorithms assume the pass/fail test is deterministic and monotone; an AI pipeline violates both assumptions, which is why the “≥3 trials per cell” rule is not optional — without it the binary search chases noise and reports a confident wrong culprit.

The method: the one-hour procedure in four quarters

Budget the hour visibly; each quarter has an exit artifact:

1. Quarter 1 (0–15): assemble and specify. Pull the Chapter 52 record: hashes, pins, retrieval snapshot, parameters, segments. Reconstruct what the ten-minute freeze left UNKNOWN. Specify the failing property and the success criterion (e.g., “cites current policy section on 5/5 probe queries”). Write H1/H2/H3 with distinct predicted stage signatures before touching anything.
2. Quarter 2 (15–30): bisect input vs. environment. Swap one side at a time: frozen pre-refresh index + current prompt vs. current index + frozen prompt, parameters fixed. Run each cell ≥3 trials. The side the failure follows owns the next quarter. If neither side cleanly owns it, H3 (interaction/fragility) leads — stated, not forced.
3. Quarter 3 (30–45): minimize to the divergence. Within the owning side, remove-until-break: narrow retrieved sets, shorten history, strip prompt lines one at a time (Ch30–31 discipline). One variable per run, predictions pre-written, distributions recorded. End with the minimal failing case and the first divergence named with hashes.
4. Quarter 4 (45–60): validate a repair candidate, queue the artifact. Apply one repair to the isolated divergence; verify on a 5–12 item mini-suite (fails pre-fix shown, passes post-fix shown, ≥3 trials on the key cells). Queue the regression test and the handoff note — merged if CI allows, queued with the record if not. Ship only the single-variable repair; everything wider goes to the full investigation.

    flowchart TD
    Q1["Q1 0-15: assemble the Ch52 record + reconstruct UNKNOWN pins; specify the success criterion; write H1/H2/H3"] --> Q2["Q2 15-30: bisect input vs environment — swap one side at a time (frozen index + current prompt / current index + frozen prompt), >=3 trials per cell"]
    Q2 --> O{"which side does the failure follow?"}
    O -->|"input / retrieval side"| H1["H1 retrieval regression"]
    O -->|"assembly side, fixed index"| H2["H2 assembly interaction"]
    O -->|"neither side cleanly owns it"| H3["H3 prompt fragility / interaction — stated, not forced"]
    H1 --> Q3["Q3 30-45: minimize within the owning side — remove-until-break by structural level, one variable per run; name the first divergence with hashes"]
    H2 --> Q3
    H3 --> Q3
    Q3 --> Q4["Q4 45-60: apply ONE repair; verify on a 5-12 item mini-suite (red pre-fix, green post-fix, x3 key cells); QUEUE the regression test + handoff — the last 10 minutes are writing"]
  

ONE-HOUR RECORD (isolation with trials):
case: ___ criterion: ___ | pins: model ___ index ___ params ___ (UNKNOWN: ___)
bisection: input-side ___/___ vs env-side ___/___ (x3 each) -> owns: ___
minimized: first divergence ___ at ___ (hash ___) | H1/H2/H3: ___ / ___ / ___
repair: ___ mini-suite pre ___/___ red, post ___/___ green (x3 key cells)
QUEUED: test-___ + handoff ___ | KEPT: bisection+trials+minisuite | SKIPPED: calibration, ledger, postmortem, generality
RULE: unqueued evidence evaporates. The last ten minutes are writing, not probing.

OBSERVATION (constructed illustration, not a measured run): bisection shows the failure following the index side 3/3 (pre-refresh snapshot passes, current fails under the fixed prompt); minimization narrows it to policy-section chunks missing from the top-k after the refresh. UPDATED BELIEF: H1 supported for this instance (retrieval regression on policy sections); H2 exonerated here (assembly byte-identical); H3 noted as residual risk (prompt lacks a missing-citation policy — queued, not fixed this hour). One repair ships; one fragility queues.

No paraphrased retrieval (“chunks looked fine”), no confidence in the repair, no agreement across two retries, no single green rerun, and no downstream quiet (“flags stopped”) substitutes for bisected trials and a bidirectional mini-suite.

Example: the stale-policy hour, quarter by quarter

The practitioner works the handoff with the clock visible:

# standard-depth isolation: bisect, minimize, validate (one variable throughout)
rec = assemble_record(handoff.freeze)  # Q1 OBSERVATION: full pins, segments, snapshots
# Q2 bisection: swap one side, predictions pre-written, x3 trials per cell
for env in [pre_refresh_index, current_index]:  # prompt + params fixed
    for t in range(3):
        log(env, trial(t, rec.input, env))  # MEASUREMENT: failure follows current index 3/3
# Q3 minimization: narrow retrieved sets until the divergence appears/disappears
minimal = minimize_retrieval(rec.input, current_index)  # first divergence: policy chunks absent top-k
# Q4 repair candidate: single variable (retrieval depth 6->12 on policy queries), mini-suite both directions
verify_both_directions(repair, failing_case=minimal, suite=mini_suite(8), trials=3)
queue(test="test-52", handoff_note=record)  # last ten minutes: writing

In the constructed case the depth repair passes the mini-suite 8/8 pre-to-post while the prompt-fragility note queues separately — the hour ships one thing and names the next thing, which is exactly its contract. The licensed claim covers this defect under these pins and this suite — not retrieval debugging in general.

Lab 58: hour-boxed isolation with pre-written bisection predictions (proposed)

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own timed hour.

Setup. Take one real defect with log access and pinnable environment (or a staged equivalent with two index snapshots). Set a visible sixty-minute timer with quarter alarms. The isolation target (input-side vs. environment-side ownership) is the independent variable; defect, tooling, and suite are controlled.

Task.

1. Before starting, write H1/H2/H3 with distinct predicted bisection signatures and the minimization path each predicts.
2. Execute the four quarters; run every cell ≥3 trials; record OBSERVATION (per-trial verdicts verbatim) and UPDATED BELIEF per hypothesis.
3. Reserve the last ten minutes for writing: the record below must be complete when the timer ends, queued artifacts named.

Success criterion. A complete one-hour record with bisection trials, a minimized divergence with hashes, a bidirectional mini-suite verdict, and queued artifacts — inside sixty minutes. An unqueued “finding” or a two-variable repair is explicitly not completion.

Companion tool: One-Hour Investigation Checklist

What it accepts: the ten-minute handoff (or direct case spec), the assembled per-request record with pins, bisection trial logs with pre-written predictions, the minimization trail, and the mini-suite bidirectional verdicts. What it performs: it verifies the case was specified with a success criterion, exactly one variable moved per run, every cell ran the declared trials, the divergence is minimized with hashes, the repair candidate shows both directions on the suite, and artifacts are queued — refusing isolation claims built on single runs or two-variable swaps. What it can establish: whether the divergence is isolated to its owning side and whether the repair candidate holds on the mini-suite — for the examined case, pins, and suite only. What it cannot establish: calibration, cost optimality, cross-incident generality, or certification. It never treats paraphrase, confidence, agreement, single runs, scores, or quiet as isolation evidence. How its output changes your next action: isolated-and-queued routes to merge plus scheduled postmortem; isolated-unqueued routes to writing before anything else; unisolated routes to the full investigation (Ch59) with the hour’s record as its opening artifact — each as one queued item.

Paper form, sufficient for this chapter:

Case ___ criterion ___ | Bisection: input ___/___ env ___/___ (x3) owns ___
Minimized: ___ (hash ___) | H1/H2/H3: ___ / ___ / ___
Repair ___: pre ___/___ red post ___/___ green | QUEUED: test-___ handoff ___
KEPT: ___  SKIPPED (named): ___

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. Isolate, then queue.

Research lineage: delta debugging under noise, and the isolation/certification gap

Two algorithms, both used here. Zeller and Hildebrandt’s delta debugging comes in two forms. ddmin takes one failing input and simplifies it until removing any remaining element makes the failure disappear — Quarter 3’s remove-until-break. dd takes a passing and a failing configuration and isolates the minimal set of differences that flips the outcome — Quarter 2’s input-vs-environment swap, where the “change set” is {index snapshot, prompt version, parameters}. Zeller’s later Isolating Cause-Effect Chains (ESEC/FSE 2002) extends dd to program state (variable values at a point), the direct ancestor of activation-level bisection in earlier chapters. Revision-control bisect is dd applied to commit history — the same binary search over a linearly ordered change set.

The determinism assumption is the catch. Both algorithms model the test as a deterministic function from configuration to {pass, fail, unresolved}, and dd’s efficiency depends on the outcome being roughly monotone in the change set. An LLM pipeline breaks both: the same configuration yields pass on run 1 and fail on run 3, and adding a chunk can fix or break depending on interaction. The chapter’s response — ≥3 trials per cell, distributions recorded, “unresolved” treated as a real third outcome rather than forced to pass/fail — is the minimal adaptation that keeps delta debugging sound on a stochastic system. This is the same batch-nondeterminism concern from earlier chapters seen from the isolation side.

Minimization has structure. Plain ddmin treats the input as a flat list of atoms; hierarchical delta debugging (Misherghi & Su, ICSE 2006) runs the same reduction level by level over a tree — drop whole sections before sentences, whole retrieved documents before chunks, whole tool-call arguments before fields — pruning large irrelevant regions early and keeping every candidate syntactically valid, so a malformed intermediate reads as a genuine “unresolved” rather than a spurious pass. An AI pipeline’s minimization targets are tree-shaped (system/task/guardrail prompt blocks, a ranked document list, structured tool arguments), so Quarter 3 minimizes by structural level, not by character span. Domain-specific reducers such as C-Reduce (Regehr, Chen, Cuoq, Eide, Ellison & Yang, PLDI 2012) enforce the same discipline from the other side: the reduction must preserve validity, and its “interestingness test” — here the success criterion plus the ≥3-trial repeat — is what stops the reducer from converging on a minimal input that fails for a different reason than the original.

Isolation is not certification — and that is a known boundary. Delta debugging isolates a minimal failure-inducing difference; it does not prove that difference is the cause in any deeper sense, and Zeller is explicit that the result is relative to the test, the configuration space, and the granularity of the change set. The chapter’s “the hour isolates but does not certify” is that caveat restated: a different change-set decomposition or a wider configuration space could isolate a different difference, so the hour’s output is a well-supported isolated divergence with a queued test, not a closed case.

Bounds: delta debugging is defined for code and structured inputs; its transfer to retrieval snapshots and prompt versions is by analogy of change-set structure, sound only when the pieces are independently swappable and the test is repeated; HDD and C-Reduce are for source-code trees, and the “minimize by structural level” transfer assumes the prompt/document/argument tree is well-formed. The transferable core: bisect the difference, minimize the input by structural level, budget the trials that a noisy oracle demands, and inherit the “relative to the configuration space” caveat honestly.

Reusable procedure: the one-hour investigation

1. Assemble — full record, pins, success criterion, H1/H2/H3 predicted (Q1).
2. Bisect — input vs. environment, one side per cell, ≥3 trials (Q2).
3. Minimize — remove-until-break within the owning side (Q3).
4. Validate + queue — one repair, bidirectional mini-suite, artifacts queued (Q4).
5. Name the skips — calibration, ledger, postmortem, generality explicitly deferred.

Failure modes

• Trial skipping. “No time for repeats” producing provisional isolation announced as final. Budget trials first, probes second.
• Two-variable swaps. Saving a round, destroying attribution. The hour’s most expensive shortcut.
• Suite-free shipping. Repair on the instance alone. Instance fixed, class scheduled.
• Calibration from anecdotes. Thresholds from three samples. Queued to Chapter 54’s procedure, never set here.
• Quarter-4 probing. New investigations at minute 50. The last ten minutes belong to writing.
• H3 forcing. Declaring a clean owner when neither side owns it. Interaction hypotheses are valid verdicts.
• Artifact evaporation. Findings without queued tests. The hour happened; nothing persists.

Limits, per contract: one hour covers one defect under one pin set with a mini-suite; it isolates but does not certify; thresholds, ledgers, postmortems, and generality wait for deeper budgets. UNKNOWN wherever pins, trials, or suite coverage were absent — named on the record.

References

• Andreas Zeller and Ralf Hildebrandt. Simplifying and Isolating Failure-Inducing Input. IEEE Transactions on Software Engineering, 28(2), 2002, pp. 183–200. https://doi.org/10.1109/32.988498
• Andreas Zeller. Isolating Cause-Effect Chains from Computer Programs. Proceedings of the Joint ESEC/FSE Conference, 2002. https://doi.org/10.1145/587051.587053
• Andreas Zeller. Why Programs Fail: A Guide to Systematic Debugging (2nd ed.). Morgan Kaufmann, 2009. (Delta debugging, cause-effect chains, and the scientific hypothesis–trial–result loop.)
• Ghassan Misherghi and Zhendong Su. HDD: Hierarchical Delta Debugging. Proceedings of the 28th International Conference on Software Engineering (ICSE), 2006, pp. 142–151. https://doi.org/10.1145/1134285.1134307 (minimize tree-structured input level by level; keep every candidate syntactically valid).
• John Regehr, Yang Chen, Pascal Cuoq, Eric Eide, Chucky Ellison, and Xuejun Yang. Test-Case Reduction for C Compiler Bugs. Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), 2012, pp. 335–346. (C-Reduce: validity-preserving domain-specific reduction; the “interestingness test”).
• Git documentation. git-bisect. https://git-scm.com/docs/git-bisect (delta debugging applied to commit history).
• Brittany Johnson, Yuriy Brun, and Alexandra Meliou. Causal Testing: Understanding Defects’ Root Causes. Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (ICSE), 2020. https://doi.org/10.1145/3377811.3380377 (cross-ref Ch 1 — why an isolated difference still needs a causal test to be called a cause).

Debugging Checklist

• Case specified with a measurable success criterion?
• Full record assembled (hashes, pins, snapshots, segments)?
• H1/H2/H3 bisection predictions pre-written with distinct signatures?
• Input-vs-environment bisection run single-variable, ≥3 trials per cell?
• Divergence minimized with hashes (one variable per run)?
• Repair candidate verified both directions on a mini-suite?
• Artifacts queued (test ID + handoff) in the last ten minutes?
• KEPT vs. SKIPPED stated on the record?
• No paraphrase, confidence, agreement, single runs, scores, or quiet cited?

What This Chapter Established

• The one-hour investigation as standard isolation: assemble, bisect, minimize, validate-and-queue in four quarters — demonstrated on the constructed stale-policy regression, no measured runs claimed.
• The input/environment/fragility separation (H1/H2/H3) with the bidirectional mini-suite as the shipping gate.
• Lab 58 as a proposed hour-boxed drill the reader executes; the One-Hour Investigation Checklist contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: any calibration, cost, generality, or certification claim. Isolation with queued artifacts; nothing universal.
• Position in the arc: Chapter 57 triages in ten minutes; this chapter isolates in sixty. The method holds its shape across budgets by stating its skips.
• Research grounding: Quarter 2 = Zeller’s dd (isolate the minimal difference), Quarter 3 = ddmin (minimize the input), run by structural level in the hierarchical-delta-debugging sense (Misherghi & Su) on the prompt/document/argument tree; git bisect = dd over history; the ≥3-trials rule is the adaptation that keeps delta debugging sound on a nondeterministic oracle, and a validity-preserving “interestingness test” (C-Reduce) is what keeps a minimized input failing for the original reason; “isolation ≠ certification” is Zeller’s own “relative to the configuration space” caveat.

Next

The hour isolates one defect and queues one artifact — but some failures refuse the hour: multi-stage incidents with money moved, conflicting evidence across services, responders disagreeing, prevention owed to regulators or the whole user base. Those need the complete procedure with roles, artifacts, and a published record. Chapter 59, “The Full AI Incident Investigation,” states it; what full rigor requires — and what it still cannot promise — is its chapter’s to establish, not this one’s.