Part VII — Debugging Agents

Two runs, same task, opposite ledgers — diff the time, not the text

Chapters 40–41 test one run’s counterfactuals. Now the practitioner holds something more common: run P (passing — one refund, ledger clean) and run F (failing — two refunds) on the same task, same pins, same day. The final answers differ by one sentence. The tool logs differ everywhere — retries, latencies, paraphrases — and naive text-diffing buries the one step that matters under forty cosmetic ones.

OBSERVATION: runs P and F share task, model/seed, tool revisions; P terminal state h9f2-class (one refund), F terminal h7a1-class (two refunds); raw log diff shows 47 differing lines across 30 steps. HYPOTHESIS H1 (early branch): P and F part at one alignable step whose divergence explains the terminals. H2 (accumulated drift): no single parting step — small per-step differences compound with no localizable divergence. H3 (unaligned): the runs are structurally incomparable (different plan shapes), so step-diffing is unlicensed. INFERENCE: none yet — H1/H2/H3 separate only by step-aligned comparison with a pre-registered alignment rule, never by line diffs.

This chapter’s question: how do you align two trajectories step-by-step — and does the alignment name one first-divergent step or prove there isn’t one?

Why “diff the logs” fails first

The obvious move — line-diffing the two logs — fails because trajectories vary legitimately where text comparison sees failure. Four defects hide behind raw diffing:

1. Cosmetic flooding. Latencies, reworded thoughts, and paraphrased observations diff on every line while deciding fields (args, hashes, codes) match. Line diffs amplify noise and hide signal.
2. Index drift. An extra retry in F shifts all later indices by one — step 14 in P compares against step 15 in F, manufacturing false divergences down the whole tail.
3. Narrative comparison. “P’s reasoning sounds more careful.” Self-narration is behavior, not trajectory — thoughts are excluded from alignment keys by rule.
4. Single-pair conviction. One P/F pair naming the cause. Nondeterministic agents need multiple pairs before the divergence generalizes even to the task.

OPINION: line-diffing trajectories is like diffing two chess games by ink color — every move differs, nothing is learned. Align by state and action, ignore the handwriting.

The mental model: trajectory diff is step alignment on deciding fields (plan structure, action hashes, observation codes, state hashes) followed by first-divergent-step naming — the trajectory analog of Chapter 2’s first divergence. Aligned-and-matching steps are shared history; the earliest aligned step whose deciding fields differ (and stay different downstream) is the branch point. Everything after it is consequence until proven otherwise.

“Diff the failure against its passing twin” is nearest-neighbor fault localization, decades old: Renieris and Reiss select, from many passing runs, the one whose execution spectrum most resembles the failing run, then report what the failing run did that the twin did not (Renieris & Reiss, 2003). The alignment key this chapter pre-registers is their distance criterion; the first-divergent step is their reported difference. And the reason to exclude thoughts and latencies is the same reason a text diff is only as good as its notion of “a line” (Myers’s classic algorithm operates on whatever unit you feed it): choose the wrong unit and every step differs. Aligning two sequences while gapping the parts that do not correspond is global sequence alignment — the problem Needleman and Wunsch solved in 1970 (Needleman & Wunsch, 1970); the pre-registered alignment key and coverage threshold are this chapter’s version of its gap penalty, and tree-edit distance is the analogue when the H3 threshold puts whole plan shapes in question.

The method: align on structure, compare on fields, name one step

Align first, compare second — in this order, with the alignment rule pre-registered:

1. Choose the alignment key. Landmark-based (plan milestones, tool-call sequence ignoring retries/thoughts) or state-anchored (match steps with equal state hashes across runs). Pre-register which; record unalignable spans as gaps, never force-matched.
2. Align, then compare deciding fields only. Per aligned pair: action hashes, observation codes/flags, state hashes, consumed lists. Thoughts, latencies, and prose excluded from comparison by rule (latency enters only for H-storm spans per Chapter 39).
3. Name the first-divergent step. Earliest aligned pair differing on a deciding field with downstream persistence (later aligned pairs also differ or terminals differ in the predicted direction). Transient differences that reconverge are noted, not convicted.
4. Test H3 (unaligned). If <50% of steps align under the pre-registered rule (threshold pre-registered; changeable setup choice, not a universal constant) or plan structures differ irreconcilably, the verdict is H3 — no step named, no cause claimed.
5. Replicate across pairs. ≥2 additional P/F pairs (≥3 total) under the same pins; a branch point that does not recur is branch-point-for-those-runs, not the defect.

    flowchart TD
    PR["pre-register the alignment key (landmark or state-anchored) and the coverage threshold"] --> AL["align P and F structurally; record unalignable spans as gaps, never force-matched"]
    AL --> C{"alignment coverage >= threshold and plan shapes reconcilable?"}
    C -->|no| H3["H3 unaligned — no step named; route to plan-shape review (Ch38)"]
    C -->|yes| CF["per aligned pair: compare deciding fields only (action hash, obs code/flags, state hash, consumed list)"]
    CF --> FD["name the first aligned pair that differs with downstream persistence"]
    FD --> RP["replicate across >=3 P/F pairs, same pins"]
    RP --> R{"same aligned branch step recurs?"}
    R -->|">=2/3 pairs, same direction"| H1["H1 branch point — hand the address to the Ch41 causal test"]
    R -->|"scattered, no persistent step"| H2["H2 accumulated drift — widen pairs, check pins / snapshots"]
  

STEP-ALIGNED DIFF (constructed; landmark alignment on tool sequence):
P-4/F-4 MATCH   | act hash 9c2e/9c2e | obs ok(r-991)/ok(r-991) | state h3f..
P-5/F-5 MATCH   | summarize; state hashes equal (h4aa/h4aa)
P-6/F-6 DIVERGE | P consumes [r-991@4] then verify_ledger; F consumed []
                |   omits r-991, re-issues issue_refund -> FIRST DIVERGENCE
P-7../F-7..     | downstream persists: P terminal h9f2 (one refund),
                |   F terminal h7a1 (two) — consequence span unless re-tested
RULE: thoughts and latencies excluded; the consumed-list gap at aligned
step 6 is the finding. Raw line-diff's 47 differences collapse to 1.

OBSERVATION (constructed illustration, not a measured run): aligned steps 1–5 match on all deciding fields; aligned step 6 differs on the consumed list with downstream terminal persistence; thoughts/latencies differ throughout and are excluded by rule. UPDATED BELIEF: H1 supported for this pair (branch at aligned step 6); H2 live-or-exonerated pending the ×3-pair replication (does step 6 recur?); H3 exonerated here (28/30 steps aligned under the pre-registered rule).

Example: diffing the refund pair with an alignment sketch

The practitioner resists the 47-line diff and aligns structurally:

# trajectory diff: align on structure, compare deciding fields (no verdict yet)
A = align(P, F, key="tool_sequence_ignore_thoughts_retries")  # OBSERVATION
# per aligned pair: compare act hashes, obs codes/flags, state hashes,
# consumed lists. Thoughts/latencies excluded. Gaps recorded, never forced.
for pair in A.pairs:
    compare(pair, fields=["act_hash", "obs_code", "state_hash", "consumed"])
first = earliest_persistent_difference(A)  # candidate first-divergent step
# H3 check: alignment coverage < threshold or plan shapes irreconcilable
# -> UNALIGNED verdict, stop. Else replicate: two more P/F pairs, same pins.
# Predictions pre-written: "aligned step 6 consumed-gap recurs in ≥2/3 pairs."

In the constructed case three pairs reproduce the aligned-step-6 consumed gap (3/3 pairs, same direction, same terminal classes): the branch point generalizes to the task under these pins — still not to tasks in general, and still a branch description rather than a causal proof (Chapter 41’s machinery owns causation; diffing proposes its address).

No confidence comparison (“P sounded surer”), no judge preferring P’s prose, no agreement across the three pairs counted as extra credit beyond recurrence itself, and no downstream symptom (“P’s customer renewed”) enters the alignment. Fields align; adjectives do not.

Research lineage: one pair is a lead, many pairs are a diagnosis

Recurrence across pairs is statistical bug isolation. Liblit and colleagues evaluate Boolean predicates over program state across many failing and non-failing runs and use statistics to pick the predicates most associated with failure — precisely because a single pass/fail comparison is noisy and, when several bugs are present, conflates them (Liblit et al., 2005). This chapter’s “count how many of ≥3 pairs share the same aligned branch step” is that move at trajectory grain: the branch point you can trust is the one that recurs, and pairs that branch elsewhere are hinting at a second defect, not noise to average away.

The diff proposes the address; Chapter 41 decides the cause. Zeller’s cause-effect-chain work (Chapter 1) is the model: compare the passing and failing executions, find where their states differ, then intervene on each difference to see which one actually carries the failure. Trajectory diff does the first half — locate the state difference at aligned step 6 — and hands the second half to causal replay. A recurring branch point is a well-addressed hypothesis, not a verdict.

The automated counterpart is a process reward model, and it has the same limit as automated attribution. A PRM is trained to score each step of a trajectory as correct or not (Lightman et al., 2024) — the learned version of per-aligned-step comparison and of the “which step is wrong” question. Like the automated failure-attribution methods of Chapter 38, it is a useful triage signal but judges one trajectory in isolation, so it cannot do the cross-pair recurrence work; the trustworthy branch point is still the one you show recurs.

Lab 42: aligned diff with pre-written branch predictions (proposed)

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own aligned pairs.

Setup. Collect ≥3 passing and ≥3 failing runs of one task under identical pins (model/seed, tool/prompt revisions, initial state, snapshots). Freeze all six under the Chapter 37 contract. The run outcome (pass vs. fail) is the independent variable; task, pins, and environment are controlled. Pre-register the alignment key and coverage threshold before comparing.

Task.

1. Before aligning, write H1/H2/H3 with distinct predicted patterns: H1: “one aligned step differs on deciding fields with downstream persistence, recurring ≥2/3 pairs”; H2: “differences scattered with no persistent step”; H3: “coverage below threshold or plan shapes irreconcilable.”
2. Align ≥3 P/F pairs; record per-pair first-divergent candidates with deciding-field values verbatim.
3. Score recurrence: how many pairs share the same aligned step and direction.

Success criterion. Aligned per-pair tables with coverage scores plus a recurrence count for the candidate step. A raw log diff or single-pair branch claim is explicitly not completion.

Companion tool: Good/Bad Run Diff

What it accepts: contracted P/F run sets (≥3 pairs), the pre-registered alignment key and coverage threshold, and the predicted branch step with its direction. What it performs: it aligns each pair structurally, compares deciding fields per aligned step (excluding thoughts/prose by rule), names per-pair first-divergent candidates with persistence checks, scores alignment coverage, blocks branch claims below threshold (H3), and counts recurrence across pairs. What it can establish: whether the pairs align, where each pair first diverges, and whether one branch step recurs across pairs — for the examined task and pins only. What it cannot establish: causation (Chapter 41’s tests own that), generality beyond the task/pins, or future reliability. It never treats narration, confidence, scores, pair-count-as-proof beyond recurrence, single-pair outcomes, or downstream symptoms as diff evidence. How its output changes your next action: recurring branch → Chapter 41 causal test at that step; non-recurring/persistent-less → drift handling (widen pairs, check pins/snapshots); unaligned → plan-shape investigation (Chapter 38), never a forced step blame.

Paper form, sufficient for this chapter:

Task/pins: ___  Alignment: ___ (threshold ___%)  Pairs: ___ (coverage ___)
First-divergent per pair: ___ ___ ___ (deciding fields verbatim)
RECURRENCE: ___/3 at aligned step ___  VERDICT: H1 / H2 / H3
NEXT: Ch41 at step ___ / widen pairs / plan-shape review

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. Align before blaming.

Reusable procedure: diff every failure against its passing twin

1. Pair — ≥3 P/F pairs, same task, same pins, contracted traces.
2. Pre-register — alignment key and coverage threshold before comparing.
3. Align — structure first; gaps recorded, never force-matched.
4. Compare fields — deciding fields only; name per-pair first divergence with persistence.
5. Recur — count shared branch steps across pairs before any causal language.

Failure modes

• Line-diff triage. Raw log diffs. Forty-seven cosmetic differences are not forty-seven findings.
• Index drift blindness. Comparing by raw step number across retry-shifted runs. Align structurally or manufacture divergences.
• Thought comparison. “P reasoned better.” Thoughts are behavior, excluded from alignment keys and comparisons alike.
• Forced alignment. Matching structurally different plan shapes step-for-step. Below-threshold coverage means H3, not effort.
• Single-pair causation. One aligned divergence closing the case. Recurrence across pairs first, causation via Chapter 41 second.
• Threshold amnesia. Choosing the coverage rule after seeing the alignment. Pre-register or the alignment is tailored.

Limits, per contract: one diff covers the examined pairs under one pin set and one alignment rule; branch recurrence is task-scoped; no diff proves causation. UNKNOWN wherever coverage fails, pins differ, or traces are uncontracted.

References

• Manos Renieris and Steven P. Reiss. Fault Localization With Nearest Neighbor Queries. Proceedings of the 18th IEEE International Conference on Automated Software Engineering (ASE), 2003, pp. 30–39. https://www.cs.purdue.edu/homes/xyzhang/fall07/Papers/renieris-ase2003.pdf
• Ben Liblit, Mayur Naik, Alice X. Zheng, Alex Aiken, and Michael I. Jordan. Scalable Statistical Bug Isolation. Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), 2005, pp. 15–26. https://doi.org/10.1145/1064978.1065014
• Eugene W. Myers. An O(ND) Difference Algorithm and Its Variations. Algorithmica 1(1–4), 1986, pp. 251–266. https://doi.org/10.1007/BF01840446
• Andreas Zeller. Isolating Cause-Effect Chains from Computer Programs. Proceedings of the Joint ESEC/FSE Conference, 2002. https://doi.org/10.1145/587051.587053
• Saul B. Needleman and Christian D. Wunsch. A General Method Applicable to the Search for Similarities in the Amino Acid Sequence of Two Proteins. Journal of Molecular Biology 48(3), 1970, pp. 443–453. https://doi.org/10.1016/0022-2836(70)90057-4
• Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s Verify Step by Step. International Conference on Learning Representations (ICLR), 2024 (arXiv:2305.20050). https://arxiv.org/abs/2305.20050

Debugging Checklist

• ≥3 P/F pairs, same task, same pins, contracted traces?
• Alignment key and coverage threshold pre-registered?
• Thoughts/prose excluded from alignment and comparison?
• Per-pair first-divergent step named with deciding fields verbatim + persistence?
• Coverage scored per pair; H3 honored below threshold?
• Recurrence counted across pairs (not asserted from one)?
• No narration, confidence, scores, single pairs, or symptoms cited as branch proof?

What This Chapter Established

• Trajectory diff: structural alignment plus deciding-field comparison with first-divergent-step naming — the trajectory analog of Chapter 2’s first divergence — demonstrated on the constructed P/F refund pairs, no measured runs claimed.
• The persistence and recurrence rules separating branch points from transient differences and single-pair anecdotes.
• Lab 42 as a proposed aligned-diff record the reader executes; the Good/Bad Run Diff contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: any causal claim about the branch step, any cross-task generality, or any reliability certification. Branches located; causes untested.
• Research grounding: “diff against the passing twin” is nearest-neighbor fault localization (Renieris & Reiss) and the alignment key is its distance criterion; the align-with-gaps step is global sequence alignment (Needleman-Wunsch), its gap penalty being the pre-registered key/threshold; excluding thoughts/latencies is choosing the right diff unit (Myers); recurrence across ≥3 pairs is statistical bug isolation (Liblit et al.) — the trustworthy branch point recurs, and pairs that branch elsewhere hint at a second defect; the diff locates the state difference and hands causation to Chapter 41 (Zeller’s cause-effect chains); a process reward model (Lightman et al.) is the automated per-step-scoring cousin, useful for triage but blind to cross-pair recurrence.

Next

Single-agent trajectories now diff cleanly — but production failures increasingly span agents: a planner, a researcher, and a writer that hand off context, share state, and blame each other in prose. Pairwise diffs of one agent’s runs cannot see a dropped handoff between two of them. Chapter 43, “Multi-Agent Systems,” extends the trajectory to message logs and handoff contracts and routes blame across agent boundaries; what multi-agent debugging can borrow from this Part — and what it must reinvent — is its chapter’s to establish, not this one’s.