Part VII — Debugging Agents

Same wrong answer, four different diseases

Chapters 36–37 made the trajectory readable. Now the practitioner faces the next trap: two refund-agent failures look identical — “refunded twice” — but the contracted traces convict different steps. Run A repeats issue_refund because the plan listed two refunds before any tool ran. Run B plans one refund but re-issues after a timeout it never saw. Identical ledgers, opposite repairs: fixing the plan cures A and leaves B broken; adding timeout handling cures B and leaves A double-spending.

OBSERVATION: run A plan step (index 1) lists issue_refund twice with no intervening observation; run B plan lists one refund, step 4 returns {error: timeout, charged: UNKNOWN} paraphrased as ok by the harness, step 7 re-issues. HYPOTHESIS H1 (plan-class): defect precedes all observations — the sequence was wrong before the world answered. H2 (invoke-class): the call itself was malformed (args, idempotency, retries). H3 (observe-class): the world’s answer was lost, truncated, or mislabeled. H4 (repair-class): the error was seen correctly and the recovery was wrong. INFERENCE: none yet — class assignment follows per-class step signatures, never the symptom.

This chapter’s question: given a contracted trace, which of the four classes does the first-divergent step belong to — and what same-symptom pair proves symptoms never decide?

Why “fix the symptom” fails first

The obvious move — patching whatever the final answer got wrong — fails because one symptom maps to many classes. Four confusions dominate:

1. Plan vs. invoke. A duplicated refund looks like a “retry bug” (invoke) when the plan step already contained the duplication (plan). Retry guards never fire on a plan that never retries — it executes.
2. Invoke vs. observe. A re-issued call looks like agent forgetfulness (observe) when the first call’s args carried a fresh idempotency key each attempt (invoke) — the world saw two distinct requests.
3. Observe vs. repair. A wrong recovery looks like bad error handling (repair) when the error observation was mislabeled upstream (observe) — the recovery was correct for the ok it was handed.
4. Single-story classification. One plausible narrative (“the agent is flaky with refunds”) closing all three runs. Nondeterministic agents need per-run signature classification across ≥3 trials.

OPINION: symptom-first debugging is triage by horoscope — vivid, confident, and wrong per class. Classify the step, not the story.

The mental model: four failure classes anchored to the step where the trajectory first diverges — plan (wrong sequence before evidence), invoke (wrong call), observe (wrong intake of the answer), repair (wrong recovery after a correct intake). Each class owns a distinct signature in contracted fields; the signature, not the ledger, assigns blame.

This four-class cut is a single-agent, tool-execution-granular view — one slice of a space that three different research methods carve up differently. Cemri and colleagues’ MAST taxonomy, built from over 1,600 annotated traces across seven multi-agent frameworks (annotator agreement κ = 0.88), identifies fourteen failure modes in three clusters — specification and system design, inter-agent misalignment, and task verification and termination, roughly 42/37/21% in that corpus (Cemri et al., 2025); those percentages are prevalences in seven specific frameworks, not a universal distribution. MAST’s largest cluster maps onto this chapter’s plan-class; its inter-agent cluster is the multi-agent chapters’ territory; its verification-and-termination cluster maps onto repair-class plus premature completion (Chapter 39); the chapter’s invoke and observe classes are the fine-grained tool-execution failures MAST records at coarser grain. A second cut, Who&When (Zhang et al., ICML 2025), is attribution-focused — which agent and which step — across 127 multi-agent systems, and its result is the multi-agent echo of Chapter 36’s first-divergent-step caveat: the best of three automated methods identifies the responsible agent only about 54% of the time and the exact failure step about 14% (Zhang et al., 2025). A third cut, Microsoft’s AI Red Team taxonomy, catalogs twenty-seven modes on security-versus-safety and novel-versus-existing axes, built from red-teaming rather than a trace corpus, and covers the agent-compromise, injection, and impersonation territory the four execution classes here do not (Microsoft AI Red Team, 2025). No single taxonomy is complete; the four classes below are deliberately the single-agent-execution slice.

The method: per-class signatures with same-symptom pairs

For each class, the signature states the contracted fields that must be present and the pattern that convicts. Every diagnosis must also name the same-symptom pair it rules out:

1. Plan-class. Signature: plan steps (pre-observation) already entail the failure — duplicated actions, missing verification steps, wrong order — while all observations are intact. Convicts on plan-record text vs. criterion; arg hashes and flags corroborate.
2. Invoke-class. Signature: plan correct, but the act record diverges — wrong args, missing idempotency key, wrong tool, retry with mutated args. Convicts on verbatim args + arg hashes across attempts.
3. Observe-class. Signature: act correct and world answered correctly, but the intake diverges — truncation flag, paraphrase, dropped provenance, mislabeled error. Convicts on integrity flags + consumed lists.
4. Repair-class. Signature: act, observation, and intake all correct, but the post-error steps diverge — wrong branch, repeated failing action, premature success claim. Convicts on post-error step sequence vs. the error code.

    flowchart TD
    D["first-divergent step in the contracted trace"] --> P{"do pre-observation plan steps already entail the failure (dup / missing verify / wrong order)?"}
    P -->|yes| PLAN["PLAN-class — fix plan authoring (the base-rate failure)"]
    P -->|no| I{"act record diverges: wrong args / tool / mutated idempotency key across attempts?"}
    I -->|yes| INV["INVOKE-class — fix call construction / idempotency"]
    I -->|no| O{"world answered correctly but intake diverges: truncation flag / paraphrase / mislabeled error?"}
    O -->|yes| OBS["OBSERVE-class — fix harness intake / labeling"]
    O -->|no| R{"intake correct but post-error steps diverge: wrong branch / repeated failing action / premature success?"}
    R -->|yes| REP["REPAIR-class — fix recovery policy"]
    R -->|no| U["UNKNOWN — deciding field missing or no signature matched"]
    PLAN --> N["name the same-symptom classes ruled out, citing their absent signatures; stabilize x3"]
    INV --> N
    OBS --> N
    REP --> N
  

SAME SYMPTOM ("refunded twice"), DIFFERENT CLASSES (constructed):
run A | plan@1 lists refund, refund (no obs between) -> PLAN-class
        args identical, flags clean, consumed complete; fix plan, not retries
run B | plan@1 lists one refund; act@4 idem_key k-41, act@7 idem_key k-77
        (distinct keys, same order/amount) -> INVOKE-class; fix idempotency
run C | act correct; obs@4 {timeout, charged UNKNOWN} relabeled ok by
        harness; step 7 consumes ok -> OBSERVE-class; fix harness labeling
run D | obs@4 {error: insufficient_funds} consumed correctly; steps 5-7
        retry identical call 3x then claim success vs exit nonzero
        -> REPAIR-class; fix recovery policy, not plumbing
RULE: the ledger never appears in any signature. Steps classify; symptoms corroborate.

OBSERVATION (constructed illustration, not a measured run): runs A–D share the ledger symptom and separate cleanly on plan text, arg hashes, integrity flags, and post-error sequences respectively. UPDATED BELIEF: class assigned per run for these instances; any run missing its deciding field reverts to UNKNOWN for that class (Chapter 37’s rule inherits here).

Example: classifying the timeout re-issue with a signature sketch

Run B’s practitioner resists the “forgetful agent” story and checks signatures in class order (plan → invoke → observe → repair — cheapest exoneration first):

# class assignment: signatures in order, fields verbatim (no repair yet)
plan = plan_steps(trace)  # OBSERVATION: does any pre-obs step entail the failure?
if repeats_in_plan(plan): verdict = "PLAN-class"  # run A pattern
elif arg_hashes(trace, 4, 7) differ on idem_key: verdict = "INVOKE-class"  # run B
elif integrity_or_provenance_gap(trace, 4, 7): verdict = "OBSERVE-class"  # run C
elif post_error_divergence(trace, 4): verdict = "REPAIR-class"  # run D
else: verdict = "UNKNOWN"  # missing deciding field or no signature matched
# Same-symptom pair recorded: "this run's signature is B-shaped; A/C/D
# shapes checked and absent (fields cited)." Predictions pre-written.

In the constructed case run B convicts INVOKE-class: plan lists one refund, both observations intact with truncated: no, but the two attempts carry distinct idempotency keys — the payment backend correctly treated them as distinct requests. The repair (pin one idempotency key per logical refund) is licensed for this harness revision only after ≥3 contracted re-runs reproduce the key-mutation signature.

No confidence on any generation, no judge score preferring one narrative, no agreement across the trials counted as confirmation, and no downstream symptom (“only one customer complained”) modifies the class. Hashes and flags classify; everything else comments.

Research lineage: plan-class is the high-prior failure

LLMs are weak planners, so checking the plan first is not just cheap — it is where the defects concentrate. Valmeekam and colleagues’ PlanBench evaluated LLMs as autonomous planners on Blocksworld, a deliberately simple domain, and found GPT-4 producing valid plans only a small fraction of the time (in the low tens of percent depending on prompt and format), with characteristic failures of hallucinated action preconditions and misordered steps — and models were also poor at verifying whether a given plan was valid (Valmeekam et al., 2023). MAST’s finding that specification and system-design issues are the single largest failure cluster is the same result at deployment scale. Plan-class is the base rate; the class walk starts there for a reason.

The invoke/observe/repair split matches independent taxonomies. TRAIL’s three error categories — reasoning, execution, planning (Chapter 37) — overlap this chapter’s classes: execution errors are invoke/observe, planning errors are plan, reasoning errors span plan and repair. Who&When’s annotations, from a third corpus, land on adjacent cuts again. Several taxonomies built from different trace corpora converging on adjacent cuts is weak evidence the cuts are real — and Who&When’s finding that automated attribution barely beats chance at the step level is the argument for the disciplined manual signature walk this chapter specifies, not against localizing the step.

Lab 38: same-symptom separation with pre-written class predictions (proposed)

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own classified pair.

Setup. Collect (or construct by harness fault-injection) two failing runs with the same external symptom but different suspected classes — e.g., a plan-duplicated refund and an idempotency-mutated refund. Freeze traces under the Chapter 37 contract, plus tool/model revisions. The suspected class is the independent variable; symptom, task, and environment are controlled.

Task.

1. Before classifying, write H1–H4 signatures with the exact deciding fields: H1 plan text; H2 arg hashes; H3 flags + provenance; H4 post-error sequence vs. error code.
2. Classify each run blind to the other’s label; record the same-symptom pair explicitly.
3. Re-run each run’s minimal prefix ≥3 times; record per-run class stability.

Success criterion. Two contracted traces with different convicted classes under one shared symptom plus per-run ×3 stability results. A single classified run or a symptom-matched story is explicitly not completion.

Companion tool: Agent Failure Taxonomy Checklist

What it accepts: the contracted trace, the success criterion with its external check, and the candidate same-symptom pair. What it performs: it walks plan → invoke → observe → repair signature checks in order, cites the deciding field values verbatim, records the ruled-out same-symptom classes with their absent signatures, marks field-missing classes UNKNOWN, and requires ×3 class-stability trials before a class verdict. What it can establish: which class the first-divergent step belongs to, which same-symptom alternatives the records exclude, and where the matching repair layer sits — for the examined runs only. What it cannot establish: the deeper cause inside the class (prompt wording vs. harness bug vs. model tendency needs Chapters 39–41 probes), generality across tasks, or future reliability. It never treats self-narration, confidence, scores, agreement, single-run outcomes, or downstream symptoms as classifying evidence. How its output changes your next action: PLAN routes to plan-authoring discipline; INVOKE to call-construction/idempotency fixes; OBSERVE to harness intake fixes; REPAIR to recovery-policy fixes; UNKNOWN to instrumentation — each as a single-variable intervention with pre-written predictions.

Paper form, sufficient for this chapter:

Symptom: ___ (external check ___)  Runs: X ___ Y ___ (same symptom claimed)
Class X: ___ (deciding field ___ = ___)  Class Y: ___ (deciding field ___ = ___)
Ruled out: ___ (absent signature ___)  STABILITY ×3: X ___ Y ___
NEXT REPAIR LAYER: plan / invoke / observe / repair / instrument

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. Signature before surgery.

Reusable procedure: classify every trajectory before repairing it

1. Freeze pair — same symptom, two contracted traces, revisions pinned.
2. Walk classes in order — plan, invoke, observe, repair — citing deciding fields verbatim.
3. Name the pair — which same-symptom alternative each signature excludes.
4. Mark UNKNOWN — any class whose deciding field is missing stays open, never guessed.
5. Stabilize ×3 — fixed everything, class recorded per trial before any repair.

Failure modes

• Symptom matching. “Both double-refunded, same bug.” Symptoms never classify; signatures do.
• Narrator classification. “The agent said it retried.” Self-narration is behavior, not trajectory — plan text and hashes outrank the apology.
• Skipped classes. Jumping to repair policy before checking plan and args. Walk all four in order; cheapest exoneration first.
• Single-run classing. One trace convicting a class. Nondeterminism needs three trials per run.
• Multi-layer repair. Fixing plan, idempotency, and recovery after one failure. One class, one intervention, pre-written prediction.
• Score-based sorting. Using confidence or judge scores to pick the class. Scores never appear in any signature.

Limits, per contract: one classification covers the examined runs under one revision set; it does not explain intra-class causes, does not certify any layer, and does not transfer across tasks. UNKNOWN wherever deciding fields are missing or paraphrased.

References

• Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why Do Multi-Agent LLM Systems Fail? International Conference on Machine Learning (ICML), 2025 (arXiv:2503.13657). https://arxiv.org/abs/2503.13657
• Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, and Qingyun Wu. Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025 (arXiv:2505.00212). https://arxiv.org/abs/2505.00212
• Microsoft AI Red Team. Taxonomy of Failure Modes in Agentic AI Systems. Microsoft, 2025. https://www.microsoft.com/en-us/security/blog/2025/04/24/new-whitepaper-outlines-the-taxonomy-of-failure-modes-in-ai-agents/
• Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change. Advances in Neural Information Processing Systems 36 (NeurIPS), 2023. https://arxiv.org/abs/2206.10498
• Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, and Rebecca Qian. TRAIL: Trace Reasoning and Agentic Issue Localization. arXiv:2505.08638, 2025. https://arxiv.org/abs/2505.08638

Debugging Checklist

• Contracted traces frozen (six fields per step, revisions pinned)?
• Same-symptom pair named with shared external check?
• All four class signatures checked in order with deciding fields cited verbatim?
• Ruled-out classes recorded with their absent signatures?
• Field-missing classes marked UNKNOWN (not guessed)?
• Each run re-run ≥3 times with per-trial class recorded?
• No narration, confidence, scores, agreement, single runs, or symptoms cited as classifier?

What This Chapter Established

• The four-class taxonomy (plan / invoke / observe / repair) with per-class contracted-field signatures and the class-walk order — demonstrated on the constructed runs A–D, no measured runs claimed.
• The same-symptom/different-class method proving symptoms never classify, with the ruled-out-pair recording rule.
• Lab 38 as a proposed same-symptom separation record the reader executes; the Agent Failure Taxonomy Checklist contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: any intra-class cause, any layer-generality claim, or any reliability certification. Runs classified; nothing universal.
• Research grounding: the four classes are a single-agent, tool-execution-granular cut of a space three research methods carve differently — MAST (14 modes / 3 clusters from 1,600+ traces of 7 frameworks, ~42/37/21% in that corpus — Cemri et al.), Who&When (attribution across 127 systems; automated methods ~54% agent / ~14% step — Zhang et al.), and Microsoft’s red-team taxonomy (27 modes, security/safety × novel/existing); no single taxonomy is complete. Plan-class is the base rate because LLMs are weak planners and plan-verifiers (PlanBench — Valmeekam et al.) and MAST’s largest cluster is specification/system-design; TRAIL’s reasoning/execution/planning categories land on adjacent cuts; and Who&When’s near-chance automated step attribution is the case for the disciplined manual signature walk.

Next

Classification tells you where the defect lives — but one class keeps escaping static signatures: the agent that never converges, retrying, reformulating, and re-issuing until the budget dies. Its trace classifies differently per window yet fails identically every run. Chapter 39, “Loops, Thrashing, and Retry Storms,” gives non-progress its own mechanics: progress metrics, repetition detection, and budget guards; what those mechanics cost in false halts is its chapter’s to establish, not this one’s.