Part VII — Debugging Agents

Three agents, one failure, zero owners

Chapters 36–42 debugged one agent’s trajectory. Now the practitioner inherits a support pipeline with three: a triage agent classifies the ticket, a researcher agent gathers policy evidence, a action agent issues the refund. The ledger shows the familiar double-refund — and each agent’s local trace looks clean. Triage passed the right category; research cited the right policy; action executed what it received. The defect lives in none of the trajectories and in all of the gaps between them.

OBSERVATION: triage→research handoff message contains order=8841 but no ticket scope; research returns two policy citations (one per inferred scope) with no scope label; action consumes both citations as two independent refund authorizations. HYPOTHESIS H1 (handoff loss): the contract between agents dropped the field the next agent needed. H2 (protocol violation): the message arrived complete but the receiver violated the consumption protocol. H3 (shared-state race): both agents read/wrote one shared store without coordination, and the interleaving produced the double authorization. INFERENCE: none yet — H1/H2/H3 predict different message-log and store-record signatures and separate only across agent boundaries, never inside one trace.

This chapter’s question: when every local trajectory passes and the system fails, which boundary record convicts — and which agent (if any) owns the repair?

Why “blame the last agent” fails first

The obvious move — punishing the action agent that issued the second refund — fails because terminal position is not causation. Four defects hide behind last-touch blame:

1. Handoff blindness. Each agent debugged alone, messages unread. The dropped scope field is visible only in the between-agent log nobody kept.
2. Prose protocols. Handoff “contracts” written as suggestions (“please include context”) — behavior requests, not machine-checked schemas. Suggestions do not bind; schemas do.
3. Narrator blame-routing. Each agent’s summary blames another (“research sent confusing evidence”). Self-narration is behavior, not trajectory — message records outrank all three apologies.
4. Single-run scapegoating. One interleaving convicting one agent. Concurrent agents need repeated interleaving trials before any routing verdict.

The “zero owners” problem is real and unsolved automatically. Zhang and colleagues built the Who&When dataset — failure logs from 127 multi-agent systems annotated with the responsible agent and decisive step — and found that the best automated attribution method identified the right agent only 53.5% of the time and the right step only 14.2%, with some methods below chance and frontier reasoning models no better (Zhang et al., 2025). Attribution by hand, from boundary records, is the reliable path.

OPINION: a multi-agent system without message logs is three suspects vouching for each other in separate rooms. Log the hallway, not just the rooms.

The mental model: the system trajectory is local trajectories plus boundary records — timestamped messages with schemas plus shared-store revisions with writers. Handoff failures drop/duplicate/mislabel fields between agents (H1); protocol failures consume complete messages incorrectly (H2); shared-state failures interleave uncoordinated reads/writes (H3). Blame routes by boundary signature to the contract, the consumer, or the coordination — never by terminal position.

The method: message logs as evidence, handoff contracts, blame routing

Extend the Chapter 37 contract across boundaries with three additions, then route:

1. Boundary message log. Every inter-agent message recorded verbatim with sender, recipient, timestamp/sequence, schema version, and content hash. Paraphrased or unlogged messages → boundary UNKNOWN.
2. Handoff contract per edge. Machine-checked schema per sender→recipient edge: required fields, types, idempotency semantics (“one authorization per ticket-scope”), and rejection behavior on violation (reject-and-ask vs. guess-and-proceed — pre-registered, never improvised). Since early 2025 several standardized agent-communication protocols supply this schema layer — Google’s A2A with its Agent Cards, IBM’s ACP, ANP, the OpenAI Agents SDK’s transfer_to_ handoff. Adopting one gives you a schema to check against; but the protocols are new and carry ambiguous semantics and security gaps (A Technical Taxonomy of LLM Agent Communication Protocols, 2026), so the boundary message log and the per-edge check stay the debugging instrument regardless of which protocol carries the message.
3. Shared-store journal. Every read/write with agent ID, timestamp, before/after hashes, and the message that authorized it. Unjournaled access → H3 undecidable for that span.
4. Blame routing rule. H1: message violates its schema at send (or required field absent) → sender-side contract failure. H2: message schema-valid but receiver’s consumed list or actions diverge from protocol → receiver-side failure. H3: messages valid, actions protocol-correct, but store interleaving explains the outcome across ≥2 orderings → coordination failure (no single-agent owner).

    flowchart TD
    FZ["freeze the hallway: every inter-agent message verbatim (sender, recipient, seq, schema version, hash) + the shared-store journal"] --> S{"each message valid against its sender->recipient edge schema at send?"}
    S -->|"no, field absent / mislabeled"| H1["H1 handoff loss — repair the EDGE CONTRACT (reject-and-ask), not the terminal agent"]
    S -->|yes| C{"receiver's consumed list + actions match its protocol on the valid message?"}
    C -->|no| H2["H2 protocol violation — repair the RECEIVER's consumption protocol"]
    C -->|yes| R{"replay store interleavings: does the outcome change with ordering across >=2 orders?"}
    R -->|yes| H3["H3 shared-state race — serialize / single-writer / lock (no single-agent owner)"]
    R -->|no| RE["boundary records all clean — widen to intra-agent probes (Ch38-41)"]
  

BOUNDARY RECORDS (constructed; triage->research->action):
T->R msg#3: schema v2 requires [order, ticket_scope]; sent [order=8841]
           -> SCHEMA VIOLATION at send (scope absent) => H1-shaped
R->A msg#7: two citations, no scope labels; schema v1 requires [scope_id]
           per citation -> SCHEMA VIOLATION at send (compounds msg#3)
A local:   consumed [cite_1, cite_2] as two authorizations per its
           protocol ("one authorization per citation") -> protocol-following
           on schema-violating input; last-touch blame EXONERATED here
STORE:     refunds_issued +r-991 (auth cite_1), +r-992 (auth cite_2); no race
RULE: violations at send route to the edge contract, not the terminal agent.

OBSERVATION (constructed illustration, not a measured run): both boundary messages violate their schemas at send; the action agent’s consumption follows its protocol given the violating input; the store journal shows sequential (non-racing) writes. UPDATED BELIEF: H1 supported for this instance (handoff-contract failure compounding across two edges); H2 exonerated here (receiver protocol-following on bad input); H3 exonerated here (no interleaving) — repair belongs to the edge contracts, not to any agent’s prompt alone.

Example: routing the double authorization with a boundary sketch

The practitioner freezes all three local traces plus the hallway, then routes mechanically:

# blame routing: boundaries first, agents second (no repair yet)
msgs = load_messages()  # OBSERVATION: verbatim with schema versions + hashes
for m in msgs:
    check_against_schema(m, contract(m.sender, m.recipient))  # H1 probe
for a in agents:
    check_consumption(a, msgs_to(a))  # H2 probe: valid input, deviant use?
journal = load_store_journal()  # H3 probe: interleavings across orderings
# Route: send-violation -> edge contract; valid-input-deviant-use -> receiver;
# valid-everywhere + interleaving-sensitive -> coordination. Predictions
# pre-written; each boundary re-run ≥3 times with interleavings varied.

In the constructed case the fix is contractual, not motivational: schema v3 makes ticket_scope non-nullable with reject-and-ask on violation, citation schema requires one scope_id per citation, and the action protocol changes to “one authorization per (ticket, scope) pair” with idempotency on that pair. Re-run ≥3 trials per edge-fix with fault-injected scope omissions: violations now halt-and-ask instead of double-spending in every trial. The licensed claim covers this pipeline under these schemas — not multi-agent systems in general.

No agent’s self-assessment (“I did my part correctly”), no confidence from any of the three, no agreement between two agents against the third, and no downstream symptom (“customer accepted the correction”) routes blame. Schemas, consumed lists, and journals route; prose lobbies.

Research lineage: structured handoffs are the fix, and misalignment is the biggest cluster

Handoff schemas measurably reduce cascading errors. MetaGPT’s central result is that encoding the workflow as a sequence of structured-output handoffs — each agent produces a typed artifact the next agent consumes, routed through a shared message pool — substantially cuts the “cascading hallucination” that naively chained LLM agents suffer (Hong et al., 2024). “Suggestions do not bind; schemas do” is not a slogan; it is the difference between MetaGPT and an AutoGPT-style free-text chain.

Inter-agent misalignment is the largest single-category failure. MAST (Chapters 36, 38) puts communication breakdown, context loss on handoff, conflicting outputs, and format mismatch at roughly 37% of multi-agent failures — its second cluster and, mode-for-mode, the densest (Cemri et al., 2025). H1 (handoff loss) and H2 (protocol violation) are that cluster; the boundary message log is the instrument for it.

H3 is a classic data race. The shared-store interleaving failure is the same phenomenon Chapter 9’s flaky-test taxonomy files under “concurrency” and “order dependency” — decades of distributed-systems practice say the fix is serialization, single-writer ownership, or explicit locking, not a smarter agent.

The hallway is also the attack surface. H1 and H2 have a malicious twin: an agent impersonating another, or a forged message injected across a boundary — Microsoft’s AI Red Team taxonomy (Chapter 38) catalogs agent compromise, injection, and impersonation as novel agentic failure modes, and cross-agent prompt injection is Chapter 19’s problem at a boundary. The boundary message log with sender authentication and content hashes is the instrument for that too.

Lab 43: boundary fault-injection with pre-written routing predictions (proposed)

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own boundary log.

Setup. Take a two-or-more-agent pipeline with an exportable message log (or add one per this chapter’s contract) and a verifiable side effect. Freeze agent revisions, schemas, and store state. The injected boundary fault (scope dropped at send vs. valid message with deviant consumption vs. racy interleaving) is the independent variable; agents, task, and environment are controlled.

Task.

1. Before injecting, write H1/H2/H3 with distinct predicted boundary signatures: H1: “sent message violates schema (field absent/mislabeled)”; H2: “message schema-valid, receiver consumed-list/actions diverge from protocol”; H3: “messages valid, outcome varies with store ordering across trials.”
2. Inject one fault class per trial series (≥3 trials each); record message validity, consumption, and journal orderings verbatim.
3. Route each series and record the repair layer (edge contract / receiver protocol / coordination).

Success criterion. A boundary log with schema verdicts per message, consumption checks per agent, and journal orderings plus per-fault routing with ×3-trial stability. Single-agent traces alone are explicitly not completion.

Companion tool: Multi-Agent Interaction Map

What it accepts: all local contracted traces, the verbatim boundary message log with schema versions, per-edge handoff contracts, and the shared-store journal. What it performs: it renders agents as nodes and messages/store-accesses as timestamped edges, checks every message against its edge schema at send and every consumption against receiver protocol, replays store interleavings, flags unlogged or unchecked boundaries UNKNOWN, and routes each failure span to edge / receiver / coordination with the deciding records cited. What it can establish: which boundary first diverges, which schema or protocol clause it violates, and where the repair belongs — for the examined pipeline and schemas only. What it cannot establish: intra-agent causes (Chapters 38–41 own those), generality across pipelines, or future coordination safety. It never treats agent self-narration, confidence, inter-agent agreement, single-interleaving outcomes, or downstream symptoms as routing evidence. How its output changes your next action: edge-routed → schema/contract repair with reject-and-ask policy; receiver-routed → consumption-protocol repair; coordination-routed → store locking/serialization or single-writer redesign; UNKNOWN → boundary instrumentation first — each as one intervention with pre-written predictions.

Paper form, sufficient for this chapter:

Pipeline: ___ (agents ___ edges ___ schemas ___)  Side effect: ___
Messages (n=___): violations ___ (msg ___ clause ___) | gaps ___ (UNKNOWN)
Consumption: deviant ___ (agent ___ protocol ___) | Store races: ___ (orders ___)
TRIALS ×3/fault: ___  ROUTED TO: edge / receiver / coordination / UNKNOWN
NEXT REPAIR: contract / protocol / serialization / instrument

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. Hallway before rooms.

Reusable procedure: debug every multi-agent failure at the boundaries first

1. Freeze hallway — messages verbatim, schemas versioned, store journaled.
2. Check sends — every message against its edge schema (H1).
3. Check receives — consumption vs. protocol on valid messages (H2).
4. Replay orderings — store interleavings varied across trials (H3).
5. Route once — single repair layer per series, predictions pre-written, ×3 trials.

Failure modes

• Last-touch blame. Punishing the terminal agent. Terminal position is sequencing, not causation.
• Suggestion contracts. “Include context please” as the handoff spec. Unchecked prose is not a contract; schemas with rejection behavior are.
• Room-only debugging. Three clean local traces closing the case. Clean rooms with a dirty hallway still double-spend.
• Narrator routing. Letting agents blame each other in summaries. Message records outrank all self-reports.
• Single-interleaving verdicts. One ordering convicting one agent. Vary orderings ≥3 ways before coordination claims.
• Simultaneous pipeline repair. Rewriting all agents and schemas after one failure. One boundary, one intervention, pre-written prediction.

Limits, per contract: one map covers one pipeline under one schema/revision set; routing does not explain intra-agent causes, does not certify coordination, and does not transfer across topologies. UNKNOWN wherever messages are unlogged, schemas unversioned, or store access unjournaled.

References

• Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, and Qingyun Wu. Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025. https://arxiv.org/abs/2505.00212
• Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, et al. MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework. International Conference on Learning Representations (ICLR), 2024. https://arxiv.org/abs/2308.00352
• Mert Cemri, Melissa Z. Pan, Shuyi Yang, et al. Why Do Multi-Agent LLM Systems Fail? International Conference on Machine Learning (ICML), 2025 (arXiv:2503.13657). https://arxiv.org/abs/2503.13657
• A Technical Taxonomy of LLM Agent Communication Protocols. arXiv:2606.19135, 2026. https://arxiv.org/abs/2606.19135
• Microsoft AI Red Team. Taxonomy of Failure Modes in Agentic AI Systems. Microsoft, 2025. https://www.microsoft.com/en-us/security/blog/2025/04/24/new-whitepaper-outlines-the-taxonomy-of-failure-modes-in-ai-agents/

Debugging Checklist

• All local traces contracted (six fields) and revisions pinned?
• Boundary messages logged verbatim with schemas, versions, hashes, order?
• Every message checked against its edge schema at send (H1)?
• Consumption checked against protocol on valid messages (H2)?
• Store journal complete with writers, hashes, authorizing messages (H3)?
• Each fault class trial-run ≥3 times with orderings varied?
• Blame routed to one layer with deciding records cited verbatim?
• No self-narration, confidence, agreement, single runs, or symptoms cited as router?

What This Chapter Established

• Multi-agent failure routing: boundary message logs, versioned handoff contracts with rejection behavior, shared-store journals, and the edge/receiver/coordination routing rule — demonstrated on the constructed triage→research→action double-authorization, no measured runs claimed.
• The hallway-before-rooms principle with the last-touch-blame exoneration method.
• Lab 43 as a proposed boundary fault-injection record the reader executes; the Multi-Agent Interaction Map contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: any intra-agent cause, any topology-generality claim, or any coordination-safety certification. One pipeline routed; nothing universal.
• Research grounding: automated multi-agent failure attribution is hard (best method ~53% at the agent, ~14% at the step — Zhang et al.), so boundary-record routing is done by hand; structured-output handoff schemas measurably reduce cascading errors (MetaGPT / Hong et al.) — “suggestions don’t bind, schemas do” — and since 2025 standardized protocols (A2A, ACP, ANP, OpenAI Agents SDK handoffs) supply that schema layer with their own ambiguities to instrument; inter-agent misalignment is MAST’s densest failure cluster (~37%); the boundary is also the attack surface (impersonation, cross-agent injection — Microsoft taxonomy, Ch38); H3 is a classic data race with classic fixes (serialization, single-writer, locking).
• Part VII’s closing map: Chapter 36 defined the trajectory, 37 instrumented it, 38 classified its failures, 39 broke its loops, 40 replayed it, 41 tested its causes, 42 diffed its runs, this chapter routed its boundaries. Trajectories, end to end.

Next

The trajectories are defined, instrumented, classified, de-looped, replayable, causally testable, diffable, and routable across agents — by human practitioners running disciplined procedures. The open question is whether any of that discipline can itself be delegated: can one AI system perform trajectory debugging on another without inheriting the very failure modes (narration-as-evidence, single-run verdicts, score worship) this Part forbids? Part VIII opens with that question. Chapter 44, “Can One AI Debug Another?”, asks under what conditions machine-assisted debugging stays inside the evidence contracts built here; what those conditions are — and whether they hold — is its chapter’s to establish, not this one’s.