Part VII — Debugging Agents

The log says “tool ran” — and nothing else

Chapter 36 promoted the failure to a trajectory. Now the practitioner opens the log and finds this: step 5: search_orders — ok, step 6: issue_refund — ok, step 9: done. No arguments, no returns, no state hashes, no latencies, no truncation flags. The double-refund from Chapter 36 is back, and the trace cannot separate any of its three hypotheses — the observation that would convict one of them was never written down.

OBSERVATION: 11-step export contains tool names and ok markers only; args, returns, state hashes, and latencies are absent for all 11 steps. HYPOTHESIS H1 (truncated observation): the refund confirmation arrived but was cut before the decision step. H2 (unlogged state): working state dropped the confirmation silently. H3 (unlogged args): step 7 re-issued with different args than step 4 (e.g., retried idempotency key), invisible without verbatim records. INFERENCE: none licensed — with this trace, H1/H2/H3 are indistinguishable, and the correct verdict is UNKNOWN, not a guess.

This chapter’s question: what must each step record — and what verdicts does each missing field forbid?

Why “log more” fails first

The obvious move — turning on verbose logging — fails because volume without a contract produces haystacks, not evidence. Four defects hide behind unstructured verbosity:

1. Paraphrased observations. The harness logs “refund succeeded” instead of {status: ok, refund_id: r-991}. Paraphrase destroys the exact key the next step needed.
2. Missing deltas. Actions logged, state never snapshotted — so an overwrite between steps 4 and 7 is untestable.
3. Truncation without flags. A 40 KB tool return silently cut to 2 KB; the agent “ignored” data it never received. Without a truncation flag, H1 looks like agent stupidity.
4. Clockless traces. No per-step latency or ordering guarantees — retries, timeouts, and out-of-order observations blur into one “confused agent” story.

OPINION: an uncontracted trace is a confession booth — plenty of text, no facts. Instrument the five fields or admit UNKNOWN.

The mental model: the trace is the instrument, not the exhaust. Like an oscilloscope, it has a specified sampling contract: per-step action/args, observation verbatim, state delta, latency, and integrity flags. A reading taken outside the contract is not a reading.

This model is borrowed, not invented. Google’s Dapper established the vocabulary distributed-systems tracing still uses: a span is a timestamped unit of work carrying start/end times, timing data, and key-value annotations, linked to its causal parent (Sigelman et al., 2010). An agent step is a span; the six contract fields are its annotations; the “consumed-by” edges are causal parent links. Dapper’s design goals — low overhead, application-transparent, instrument the shared libraries not the business logic — are why this chapter’s contract lives in the harness, not in a prompt asking the agent to narrate. The OpenTelemetry project’s GenAI semantic conventions are the current standardization of exactly these fields for LLM and agent calls.

The method: the per-step instrumentation contract

Every step records all six entries below. Any absent entry downgrades the spans that depend on it to UNKNOWN — the contract states exactly which verdicts are forbidden:

1. Action + verbatim args. Tool name, exact arguments, idempotency keys, retry counts. Missing → repeated-vs-distinct calls indistinguishable (H3 undecidable).
2. Observation verbatim + integrity flags. Full return or content hash plus truncated: yes/no, truncation_point, error_code. Paraphrase or silent cut → H1 undecidable.
3. State delta with hashes. Keys added/removed/overwritten with before/after hashes of working state (or explicit “stateless step” marker). Missing → H2 undecidable.
4. Latency + ordering. Per-step wall time, tool latency, and monotonic sequence IDs. Missing → timeout-vs-logic failures indistinguishable.
5. Context provenance. Which prior observations were included in this step’s inputs (IDs or hashes). Missing → consumed-by analysis impossible.
6. Model/config pin. Model ID, seed or sampling config, prompt/tool-definition revisions. Missing → replay verdicts (Chapters 40–41) unlicensed.

    flowchart TD
    S["one step, recorded at the HARNESS level"] --> F1["verbatim args + arg hashes"]
    S --> F2["observation verbatim + integrity flags"]
    S --> F3["state delta + before/after hashes"]
    S --> F5["context provenance (consumed-by)"]
    F1 --> H3{"H3 arg drift decidable?"}
    F2 --> H1{"H1 truncated / lost observation decidable?"}
    F5 --> H1
    F3 --> H2{"H2 state overwrite decidable?"}
    H1 -->|"field missing"| U["UNKNOWN — instrument the harness; do not guess, do not repair"]
    H2 -->|"field missing"| U
    H3 -->|"field missing"| U
    H1 -->|present| V["verdict from the recorded field, then re-run x3"]
    H2 -->|present| V
    H3 -->|present| V
  

CONTRACT RECORD (one per step; verbatim or UNKNOWN):
step 7 | act: issue_refund(order=8841, amount=42.00, idem_key=k-77)
  args_hash: 9c2e | obs: {status: ok, refund_id: r-992} (bytes 64/64, truncated: no)
  state: refunds_issued +r-992 (h 3fa1 -> 7b0d) | latency: 790ms | seq: 007
  consumed: [order_lookup@2, policy@3] | NOT consumed: [refund r-991@4] <- FLAG
  model: m-pinned/rev-12 | toolspec: refunds/v4
RULE: the FLAG (unconsumed r-991) is the finding. Without provenance
logging, this line cannot be written and H1 stays UNKNOWN.

OBSERVATION (constructed illustration, not a measured run): with the contract in place, step 7’s consumed list omits r-991@4 while carrying an identical idempotency-free arg set; truncation flags read no at steps 4 and 7. UPDATED BELIEF: H1 (observation never plumbed into step 7’s inputs) supported for this instance; truncation-caused H1-variant exonerated here by integrity flags; H3 live-or-exonerated pending arg-hash comparison.

Example: re-instrumenting the refund agent

The practitioner stops guessing and adds the contract at the harness level — not in the prompt (“remember to log!”), which is behavior, not instrumentation:

# harness-level step recorder (no re-running the agent yet)
def record_step(step, action, args, obs_raw, state_before, state_after, t0, t1):
    rec = {
        "args_verbatim": args, "args_hash": h(args),  # OBSERVATION: exact call
        "obs_verbatim": obs_raw, "obs_hash": h(obs_raw),  # never paraphrased
        "truncated": len(obs_raw) > BUDGET,  # integrity flag, always present
        "state_delta": diff_keys(state_before, state_after),  # added/removed/overwritten
        "latency_ms": t1 - t0, "consumed": input_refs(step),  # provenance
        "model_rev": MODEL_REV, "toolspec_rev": TOOL_REV,  # replay pins
    }
    return rec  # missing key -> span marked UNKNOWN, never interpolated
# Post-hoc pass: re-walk the Chapter 36 trajectory; every paraphrased
# observation becomes UNKNOWN; predictions per hypothesis pre-written.

In the constructed case the re-instrumented re-run (≥3 trials, all else fixed) reproduces the consumed-gap at step 7 in all three trials with truncated: no throughout: the harness delivered the observation, the agent step did not consume it. Plumbing defect localized to input construction for this instance — a statement about this harness revision, not about agents in general.

No confidence attached to either generation, no judge score, no agreement across the three trials treated as extra credit, and no downstream symptom (“ledger reconciled later”) enters the verdict. Flags and hashes decide; adjectives do not.

Research lineage: a complete trace is necessary, not sufficient

Instrumenting the six fields removes the excuse of missing data. It does not make the reading easy. Deshpande and colleagues built TRAIL, a benchmark of 148 annotated agent execution traces containing 841 real errors across reasoning, execution, and planning categories, and asked strong long-context models to localize the errors given the full trace. The best model scored about 11% (Deshpande et al., 2025). Two consequences for this chapter. First, the contract is worth enforcing precisely because the analysis is hard — you cannot afford to also be guessing at the data. Second, “hand the trace to a model and ask what went wrong” is not yet a substitute for the schema walk; the model is a weak reader of its own kind of logs.

TRAIL’s three error categories — reasoning, execution, planning — are also a preview of Chapter 38’s taxonomy, and its finding that reasoning models localize category and position better than non-reasoning models is a hint about where automated trace triage might eventually help.

Lab 37: contract-vs-exhaust with pre-written missing-field predictions (proposed)

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own contracted trace.

Setup. Take the Chapter 36 trajectory (or one fresh failing run with side-effect verification). Freeze tool revisions, model/seed, and initial state. The instrumentation level (legacy exhaust log vs. six-field contract) is the independent variable; task, model, seed, and environment are controlled.

Task.

1. Before instrumenting, write H1/H2/H3 with the missing field each needs: H1 needs integrity flags + provenance; H2 needs state deltas + hashes; H3 needs verbatim args + arg hashes.
2. Attempt all three verdicts on the legacy log; record which are UNKNOWN and why (field-level).
3. Re-run under the contract ≥3 times; record per-run flags, hashes, and consumed lists.

Success criterion. A per-step contract table with forbidden-verdict annotations on any remaining UNKNOWN spans plus per-run deciding-field results. A verbose-but-uncontracted log is explicitly not completion.

Companion tool: Trajectory Viewer

What it accepts: the contracted trace (six fields per step), the legacy log if one exists, and the success criterion with its external check. What it performs: it renders each step against the contract, marks every missing field with the verdicts it forbids, draws consumed-by edges from provenance data, surfaces truncation/timeout flags, and blocks causal claims on UNKNOWN spans. What it can establish: whether the trace supports a step-localized verdict, which fields decide each hypothesis, and where instrumentation (not repair) is the next action — for the examined trace only. What it cannot establish: the cause of a dropped observation (prompt vs. harness vs. model needs Chapters 38/41 probes), generality, or future reliability. It never treats paraphrased observations, confidence, agreement, single-run outcomes, or downstream symptoms as deciding evidence. How its output changes your next action: decided H1/H2/H3 routes to the matching Chapter 38 class; any UNKNOWN routes to harness instrumentation first — no prompt edits, no model swaps, no retries-as-repair.

Paper form, sufficient for this chapter:

Trace: ___ (contract v___)  Steps: ___  UNKNOWN spans: ___ (missing: ___)
Deciding fields: H1 [flag/prov ___] H2 [delta/hash ___] H3 [args/hash ___]
REPRO ×3: ___ ___ ___ (flags+hashes verbatim)  NEXT: repair-class / instrument

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. Contract before conclusions.

Reusable procedure: instrument every agent before debugging it

1. Pin — model, seed/config, prompt and tool-definition revisions.
2. Record six — args, observation + flags, delta + hashes, latency + seq, provenance, pins — per step, at the harness level.
3. Forbid explicitly — annotate each UNKNOWN span with the missing field and the verdicts it blocks.
4. Re-run thrice — fixed everything, deciding fields recorded verbatim.
5. Route by field — decided hypotheses to Chapter 38 classes; UNKNOWN to instrumentation, never to repair.

Failure modes

• Verbosity theater. Megabytes of paraphrased logs. Volume without the six fields is exhaust, not evidence.
• Prompt-as-logger. “Log your reasoning each step.” Self-narration is behavior, not trajectory — harness records outrank agent prose.
• Silent truncation. Cutting tool returns without flags. Every cut without a flag manufactures a false H2 (stupid agent) from a true H1 (starved agent).
• Hashless state. Claiming “state was fine” without before/after hashes. Undelta’d state is UNKNOWN state.
• Single-trace sentencing. One contracted run closing the case. Nondeterminism needs three trials minimum.
• Score substitution. Citing confidence, judge scores, or trial agreement as the verdict. Fields decide; scores decorate.

Limits, per contract: one instrumented trace covers one harness/model/state revision; it does not explain causes, does not certify the agent, and does not transfer across tasks. UNKNOWN wherever any of the six fields is missing or paraphrased.

References

• Benjamin H. Sigelman, Luiz André Barroso, Mike Burrows, Pat Stephenson, Manoj Plakal, Donald Beaver, Saul Jaspan, and Chandan Shanbhag. Dapper, a Large-Scale Distributed Systems Tracing Infrastructure. Google Technical Report dapper-2010-1, 2010. https://research.google/pubs/dapper-a-large-scale-distributed-systems-tracing-infrastructure/
• Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, and Rebecca Qian. TRAIL: Trace Reasoning and Agentic Issue Localization. arXiv:2505.08638, 2025. https://arxiv.org/abs/2505.08638
• OpenTelemetry Authors. Semantic Conventions for Generative AI Systems. OpenTelemetry Specification, 2024–. https://opentelemetry.io/docs/specs/semconv/gen-ai/

Debugging Checklist

• Harness-level recording (not prompt-requested narration)?
• All six fields present per step (args, observation + flags, delta + hashes, latency + seq, provenance, pins)?
• Every UNKNOWN span annotated with its missing field and forbidden verdicts?
• H1/H2/H3 mapped to deciding fields before re-runs?
• Contracted trace re-run ≥3 times (all else fixed), fields verbatim?
• Provenance/consumed-by edges drawn from recorded refs (not inferred)?
• No narration, confidence, agreement, single runs, or symptoms cited as verdict?

What This Chapter Established

• The per-step instrumentation contract (verbatim args, verbatim observations with integrity flags, state deltas with hashes, latency/ordering, provenance, config pins) with the missing-field-forbids-verdict rule — demonstrated on the constructed refund re-instrumentation, no measured runs claimed.
• The trace-as-instrument model separating contracted readings from exhaust logging.
• Lab 37 as a proposed contract-vs-exhaust record the reader executes; the Trajectory Viewer contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: any cause of the consumed-gap, any harness generality claim, or any reliability certification. One trace instrumented; nothing universal.
• Research grounding: the trace-as-instrument model is Dapper’s span/annotation/causal-parent vocabulary (Sigelman et al.), and its “instrument the harness, not the business logic” design goal is why the contract is not a prompt; OpenTelemetry’s GenAI conventions standardize these fields. A complete trace is necessary but not sufficient: models score ~11% at localizing errors in full agent traces (TRAIL, Deshpande et al.), so the schema walk is still yours.
• Forward constraint: Chapters 38–43 inherit this contract — every taxonomy class, loop claim, replay, diff, and multi-agent verdict in this Part presupposes contracted traces and is UNKNOWN without them.
• One trace object, several views: this contract is the canonical form for the rest of the book. Chapter 36’s five-field step is it with the replay pins moved to the run header; Chapter 45’s crash-dump trajectory slot is a frozen excerpt of a contracted trace around an incident; Chapter 52’s production “wide event” is one contracted step emitted as a single structured log line for fleet analysis. Same span, same annotations, same causal-parent edges — what changes downstream is retention and consumer, not structure.

Next

Steps are now readable — but readings without a vocabulary are just rows. The same double-refund signature (repeated action, unconsumed observation) can mean four different defects in four different trajectory locations. Chapter 38, “Agent Failure Taxonomy,” builds that vocabulary: plan, invoke, observe, and repair failure classes with per-class step signatures; which class a given signature belongs to is its chapter’s to establish, not this one’s.