# Advanced Agents From First Principles 13: How Do You Debug an Agent That Made the Wrong Decision? Add Trajectory Observability
# Book: Advanced Agents From First Principles
# Chapter URL: https://programmer.ie/books/advanced-agents-from-first-principles/13-chapter/
# Prose fingerprint (sha256, normalized): 62199e9acdb6bddec159c57d6d92697ddf0ac7c5ecf207de5e70521622ef54a1
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

An advanced agent fails.
You look at the final answer.
It is wrong.
So you inspect the prompt.
The prompt looks reasonable.
You inspect the model response.
That also looks reasonable.
But somewhere between the original request and the final result the system:
chose the wrong specialist, pruned the branch that contained the right solution, trusted a critic that was wrong, escalated to an expensive model unnecessarily, failed to escalate when it should have, retrieved stale memory, spent most of its budget exploring duplicates, accepted a weak verifier signal, retried the same strategy under a different name, or transformed a local success into a global failure. The final answer does not tell you which one happened.
Neither does a token count.
Neither does a single log line saying:
agent completed successfully That is the production problem this post solves.
Advanced agents need observability at the level where decisions are actually made.
If the architecture contains routers, planners, critics, search trees, specialists, memory, adaptive budgets and external verification, then the trace must expose those mechanisms explicitly.
Otherwise your system may be sophisticated at runtime and primitive at debugging time.
The first-principles problem A simple application trace often looks like this:
request ↓ function call ↓ database ↓ response An advanced agent may instead look like this:
request ↓ classify task ↓ route to specialist ↓ generate candidate states ↓ score branches ↓ prune branches ↓ escalate one branch ↓ critic review ↓ revision ↓ tool calls ↓ verify result ↓ final answer A conventional trace that only records model calls will miss the most important events.
The agent’s behavior is not merely a sequence of prompts.
It is a sequence of decisions about computation.
Those decisions need lineage.
Observability is not logging everything There is an easy but dangerous response to this problem:
Log everything.
That gives you enormous traces containing:
prompts, completions, embeddings, tool payloads, every branch, every token, every retrieved memory, every intermediate score. Then a production failure occurs and nobody can determine what matters.
High-volume logging is not the same thing as observability.
Useful agent observability answers specific questions:
What decision happened? Why did it happen? What alternatives existed? What evidence was available then? What state did the decision operate on? How much did it cost? What happened because of it? Was the final outcome verified? The unit of observability should therefore be the decision event.
A decision event A useful generic event can be represented like this:
from dataclasses import dataclass, field from datetime import datetime, timezone from typing import Any @dataclass class DecisionEvent: event_id: str run_id: str parent_event_id: str | None kind: str component: str state_id: str timestamp: str decision: str reason: str alternatives: list[str] = field(default_factory=list) evidence_ids: list[str] = field(default_factory=list) model: str | None = None prompt_version: str | None = None cost: float = 0.0 latency_ms: int = 0 metadata: dict[str, Any] = field(default_factory=dict) def now_iso() -> str: return datetime.now(timezone.utc).isoformat() The exact schema is not important.
The separation is.
A decision event should distinguish:
state choice reason alternatives evidence cost outcome That is enough to reconstruct much of the causal path later.
Run IDs are not enough Most systems already have a trace ID or run ID.
That identifies one execution.
But an advanced agent contains branching structure.
You need identities at several levels:
run_id ↓ trajectory_id ↓ state_id ↓ node_id ↓ event_id ↓ evidence_id These identities solve different problems.
A run_id answers:
Which user request did this belong to?
A trajectory_id answers:
Which candidate path did this action belong to?
A state_id answers:
What exact world or repository state was being considered?
A node_id answers:
Which search node did we expand, evaluate or prune?
An evidence_id answers:
What observation justified the decision?
Without these distinctions, branches from the same run become difficult to reconstruct.
State identity matters Suppose a coding agent runs tests on commit A.
It then modifies the repository to commit B.
The tests from A passed.
If your trace only says:
tests: PASS then the evidence can accidentally be associated with B.
A stronger trace says:
verification_event: state_id: git-tree:7f91... verifier: pytest result: PASS Now the evidence is bound to the exact state it tested.
This is the same principle we used in the verification post:
Evidence without state identity can become stale evidence.
Observability must preserve that relationship.
The trace is a graph, not a list Sequential logs work well when execution is sequential.
Search is not sequential.
Consider a Tree of Thoughts run:
root / | \ A B C / \ \ A1 A2 C1 \ A2a A timestamp-ordered log might show:
create A create B create C score A score B score C create A1 create A2 create C1 score A1 ... That tells you execution order.
It does not clearly tell you lineage.
Store the graph explicitly:
@dataclass class SearchNodeTrace: node_id: str parent_id: str | None trajectory_id: str state_id: str depth: int action: str score: float | None status: str reason: str | None Possible statuses might include:
GENERATED EVALUATED EXPANDED PRUNED SELECTED VERIFIED FAILED Now you can ask a much more useful debugging question:
Which ancestor decision led to the final failure?
Record why a branch was pruned A search trace that says:
node B7 -> PRUNED is incomplete.
Why was it pruned?
Possible reasons include:
score below beam threshold hard constraint violated duplicate state budget exhausted verifier failed risk policy rejected parent dominated stale state Record the reason as structured data.
For example:
@dataclass class PruneEvent: node_id: str state_id: str reason_code: str score: float | None threshold: float | None competing_node_ids: list[str] Then you can measure pruning behavior across thousands of tasks.
That matters because the benchmark post introduced pruning regret.
Observability gives you the data needed to calculate it in production.
Pruning regret needs lineage Suppose branch C was pruned early.
Later offline replay shows that C would have produced a verified solution.
You now know the search system committed a selection error.
Conceptually:
pruned branch ↓ offline continuation ↓ verified success ↓ pruning regret Without branch identity and prune reasons, this analysis becomes guesswork.
With them, you can ask:
Which scorer pruned the successful branch? At what depth? Against which competing branch? At what score margin? How often does this happen? That is a production debugging loop.
Routing decisions need the same treatment Mixture-of-experts and mixture-of-agents systems introduce another hidden decision:
task ↓ router ↓ expert If the expert fails, the router may be the real problem.
So route traces should capture:
@dataclass class RouteEvent: task_id: str state_id: str available_routes: list[str] selected_route: str scores: dict[str, float] reason: str escalation_allowed: bool This lets you distinguish:
expert failure vs routing failure That distinction is crucial.
If a coding specialist fails on a legal-research task, improving the coding specialist is irrelevant.
The route was wrong.
Routing regret The benchmarking post introduced cost-weighted routing regret.
Observability lets you compute it from actual production traces.
Suppose the router chose:
frontier_model cost = $0.40 verified = PASS But replay shows:
local_specialist cost = $0.01 verified = PASS The original route succeeded.
It was still wasteful.
Conversely:
local_model cost = $0.01 verified = FAIL when a specialist would have passed is a missed escalation.
The trace must preserve enough information to tell those apart.
Escalation should be observable Adaptive agents often use escalation policies:
cheap path ↓ uncertainty high? ├─ no -> continue └─ yes -> stronger model A production trace should explain:
why escalation happened what threshold fired what extra cost was incurred whether escalation changed the outcome For example:
@dataclass class EscalationEvent: from_policy: str to_policy: str signal_name: str signal_value: float threshold: float incremental_cost: float Then you can measure:
useful escalation rate unnecessary escalation rate missed escalation rate cost per recovered failure Without these events, adaptive systems become difficult to calibrate.
Critics need attribution Planner/executor/critic systems can fail in several ways.
The generator may be wrong.
The critic may correctly identify the error.
The reviser may ignore the criticism.
Or the critic may damage a correct answer.
A final output does not tell you which happened.
Trace the transitions:
candidate_v1 ↓ critic_1 ↓ critique ↓ candidate_v2 ↓ external verifier Record:
before score critic finding revision applied? after score external outcome Then calculate:
wrong → correct correct → correct wrong → wrong correct → wrong The last category is especially important.
A critic that frequently turns correct solutions into failures is not helping merely because its prose sounds intelligent.
Multi-agent debate needs disagreement traces A debate system adds even more intermediate structure.
For every proposition or candidate, you may have:
agent A position agent B position agent C position judge decision final revision external verification Useful observability should preserve:
initial disagreement independence of positions arguments introduced position changes judge selection final correctness Otherwise you cannot distinguish genuine error correction from social convergence.
Three agents agreeing after reading the same mistaken evidence is not diversity.
It is correlated failure.
Do not confuse hidden reasoning with operational reasoning An observability system should not depend on storing private internal model reasoning.
You do not need hidden chain-of-thought to debug the runtime.
You need structured operational decisions:
selected tool X because schema matched intent Y pruned node B because score 0.31 < beam cutoff 0.54 escalated model because verifier returned UNKNOWN twice selected candidate C because test coverage passed and latency was lower These are runtime facts.
They are much more useful for engineering than an enormous free-form reasoning transcript.
A strong design records decision summaries and evidence, not dependence on hidden model internals.
Cost is part of the trajectory Advanced agents allocate computation dynamically.
Therefore every meaningful event should expose cost.
At minimum:
input tokens output tokens model calls tool calls latency monetary cost Then aggregate by component:
router $0.002 planner $0.020 search $0.110 critic $0.025 verifier $0.004 --------------- total $0.161 Now debugging changes.
You can ask:
Why did this request cost ten times more than the median?
Perhaps:
beam search produced duplicates, MCTS over-expanded one subtree, the critic oscillated, the router repeatedly escalated, verification returned UNKNOWN and triggered retries. Cost attribution makes those mechanisms visible.
Useful cost metrics Per run, record:
total_cost model_cost tool_cost verification_cost search_cost recovery_cost Across runs, calculate:
cost per verified success cost per recovered failure cost per search node cost per useful escalation cost per accepted critique These are more informative than total token usage alone.
Latency needs critical-path attribution Parallel agents complicate latency.
Suppose three specialists run concurrently:
A: 800 ms B: 1200 ms C: 4500 ms The total work is 6500 ms.
The wall-clock contribution is closer to 4500 ms.
Observability should distinguish:
work time vs critical-path latency This matters for tree search, debate, ensemble generation and specialist fan-out.
A component can consume significant compute without increasing user-visible latency if it runs in parallel.
Or one slow branch can dominate the entire request.
Verification evidence belongs in the trace The final result should not simply say:
success = true Record the verifier evidence.
For example:
@dataclass class VerificationTrace: verifier_id: str verifier_version: str state_id: str result: str # PASS | FAIL | UNKNOWN criteria: dict[str, str] evidence_ids: list[str] latency_ms: int cost: float This lets you answer:
Which criterion failed? Which verifier version produced the result? Was the evidence current? Was the verifier deterministic? Did the agent continue after UNKNOWN? Verification is not just the last line of the trace.
It is part of the causal graph.
UNKNOWN is observable information A sophisticated agent should sometimes terminate with:
UNKNOWN That can happen when:
required evidence is unavailable, an external system is unreachable, verification coverage is incomplete, a deployment state changed during checking, the acceptance test is inconclusive. Trace the reason.
A large UNKNOWN rate might indicate:
weak verifier coverage bad environment access poor evidence collection race conditions insufficient task specification If UNKNOWN gets collapsed into FAIL or PASS, you lose that diagnostic signal.
Memory needs read and write lineage Memory introduces two important events:
memory read memory write A memory read should record:
query scope candidate IDs selected IDs scores freshness provenance A memory write should record:
source event verification status memory type scope TTL/version promotion reason Then when an agent follows stale information, you can determine whether the problem was:
bad stored memory bad retrieval bad ranking bad freshness filtering bad context assembly Without memory lineage, these failures blur together.
Tool calls need before-and-after state A tool invocation trace should not only capture arguments and output.
For side-effecting tools, record:
precondition state requested action execution result postcondition state verification result Conceptually:
before_state ↓ tool action ↓ reported result ↓ after_state ↓ verification This reveals a common agent failure:
tool returned 200 OK but the intended user outcome did not occur.
HTTP success is not necessarily goal success.
A minimal event taxonomy You do not need hundreds of event types.
A useful first version might have:
RUN_STARTED STATE_OBSERVED ROUTE_SELECTED PLAN_CREATED NODE_GENERATED NODE_SCORED NODE_PRUNED NODE_SELECTED MEMORY_READ MEMORY_WRITTEN TOOL_CALLED TOOL_RESULT CRITIQUE_CREATED REVISION_CREATED ESCALATION_TRIGGERED VERIFICATION_RUN RECOVERY_TRIGGERED RUN_COMPLETED The exact names do not matter.
Consistency does.
Separate events from payloads Large prompts, model responses, files and tool outputs can make event streams enormous.
Store references instead:
event ↓ payload_id ↓ object/blob store For example:
@dataclass class PayloadRef: payload_id: str sha256: str media_type: str size_bytes: int location: str This gives you compact traces while preserving reproducibility.
Hashing also helps detect whether two events operated on the same underlying content.
Redaction and privacy are part of observability design Agent traces may contain:
source code, customer data, credentials, proprietary documents, browser sessions, support tickets, internal policies. Do not treat traces as harmless debug text.
Design for:
redaction field-level access retention limits tenant isolation secret filtering deletability An observability system that leaks the information it observes is not production-ready.
Replay is the next level A trace helps explain what happened.
Replay helps test what would happen differently.
There are several forms.
Exact replay Reuse recorded outputs and deterministic state transitions.
Useful for debugging orchestration logic.
Model replay Rerun the same prompts against the same model version where possible.
Useful for measuring stochastic variation.
Counterfactual replay Change one decision:
use another route keep pruned node skip critic change beam width avoid escalation and continue the trajectory.
This is enormously useful for advanced agents.
It turns production traces into architecture experiments.
Counterfactual debugging Suppose a failed coding run used MCTS.
The selected path was:
root → B → B2 → B2a But branch A was pruned.
Counterfactual replay can ask:
What if A had survived one more expansion? If A then reaches verified success, the problem is likely the search policy or evaluator.
If A also fails, pruning may not be the issue.
This is much stronger than staring at the final patch.
Replay must control side effects Never casually replay production actions against production systems.
Prefer:
sandbox fixture mock service read-only mode transaction rollback container/worktree staging environment recorded tool output The replay engine should know which events are:
pure read-only reversible irreversible Counterfactual search over irreversible actions without isolation is a dangerous architecture.
Failure localization Once traces are structured, you can classify failures by stage.
For example:
GENERATION_FAILURE SELECTION_FAILURE ROUTING_FAILURE PLANNING_FAILURE EXECUTION_FAILURE MEMORY_FAILURE VERIFICATION_FAILURE RECOVERY_FAILURE BUDGET_FAILURE A failed run can have more than one label.
But even approximate localization changes engineering priorities.
If 60% of failures are routing failures, tuning the generator is probably the wrong investment.
If most failures are verification UNKNOWNs, improve evidence access.
If search frequently generates a successful branch but prunes it, improve selection.
Observability gives you the evidence to know which layer is failing.
Failure decomposition with oracle@N Suppose a search system generates ten candidates.
One would pass the external verifier.
But the selected candidate fails.
Then:
oracle@10 = success selected result = failure That is primarily a selection failure.
If none of the ten candidates would pass, it is primarily a generation failure.
This diagnostic only works if candidate identities and outcomes are traceable.
Failure decomposition with critics Suppose:
candidate A = correct critic says A is wrong revision B = incorrect The final run fails.
Without attribution you might blame generation.
With trajectory observability you identify:
correct → wrong caused by critic/revision stage That is an entirely different engineering action.
Failure decomposition with routers Suppose a task fails using general_model.
Offline evaluation shows code_specialist succeeds.
That is a routing miss.
Suppose the router selected code_specialist correctly but it still failed.
That is an expert capability failure.
Again, the final output alone cannot distinguish them.
Search-tree visualizations are useful, but secondary A graphical tree can be very helpful:
root ├── A score=.61 │ ├── A1 score=.69 SELECTED │ └── A2 score=.23 PRUNED ├── B score=.59 PRUNED └── C score=.18 PRUNED But the underlying structured events matter more than the visualization.
If your data model is correct, you can build:
tree views, timelines, flame graphs, routing matrices, cost charts, verifier dashboards, replay tools. If your data model is weak, dashboards merely make weak data prettier.
A useful run summary After a run, generate a concise structured summary:
Run: 8a31 Outcome: FAIL State: git-tree:4d2a Route: coding-specialist Search: generated: 18 nodes expanded: 7 pruned: 11 duplicate ratio: 0.28 Critique: 2 critiques 1 revision accepted Verification: tests: FAIL lint: PASS Cost: total: $0.14 search: $0.09 critic: $0.03 verification: $0.02 Likely failure stage: SELECTION_FAILURE That is far more useful than dumping every prompt first.
Observability for coding agents Coding agents have unusually strong state identities and verifiers.
Useful state IDs include:
git commit git tree hash worktree ID patch hash test environment hash Important events include:
files inspected symbols selected plan created patch generated tests run branch chosen critic findings verification state Useful debugging questions:
Why did the agent edit this file? Which evidence connected it to the task? Which alternative patch was rejected? Which test failure caused replanning? Did the final tests run against the final tree? This is where decision lineage becomes extremely powerful.
Observability for research agents Research agents need strong source provenance.
Track:
query source IDs publication dates claim IDs support relationships contradictions source quality retrieval rank A useful graph is:
claim ↓ source evidence ↓ extraction ↓ synthesis ↓ final statement When a claim is wrong, you can determine whether:
retrieval missed the right source, extraction misread it, synthesis overgeneralized, stale information outranked current evidence, verification coverage was missing. Observability for browser agents Browser agents require state snapshots around navigation and mutation.
Track:
URL/page identity DOM/state hash form state selected action tool result post-action page state A common failure is:
click reported success but the page did not transition as expected.
The trace should show both.
For destructive actions such as purchases, submissions or deletions, provenance and pre/post-condition checks become especially important.
Observability for data agents Data agents should bind actions to:
dataset version schema version pipeline run ID query hash transformation hash validation result Then a bad transformation can be traced through:
source version ↓ selected operation ↓ transformed state ↓ validation This also helps reproduce failures after underlying datasets change.
Observability for DevOps agents DevOps agents operate in consequential environments.
Track:
cluster/deployment state incident ID change request precondition checks selected remediation approval state execution result postcondition verification rollback availability For example:
restart service should not be represented as a lone action event.
You want:
why restart was selected what evidence supported it what alternatives existed whether approval was required what state changed whether health recovered whether rollback was needed That is operationally meaningful observability.
Observability for mixture-of-agents systems A mixture-of-agents runtime may combine:
router retriever planner local specialist frontier specialist search critic verifier The trace should allow a waterfall like:
request ↓ router: code task (.92) ↓ local coder ↓ verifier: UNKNOWN ↓ escalation ↓ frontier coder ↓ critic ↓ revision ↓ verifier: PASS Then cost attribution can explain exactly where the extra compute was spent.
This is essential for optimizing heterogeneous systems.
Build traces before dashboards A practical implementation order is:
1. stable IDs 2. event schema 3. state identity 4. evidence references 5. cost attribution 6. failure classification 7. replay 8. dashboards Do not begin with a beautiful UI.
Begin with data you can trust.
A tiny trace recorder A minimal implementation can be extremely small:
import json from dataclasses import asdict from pathlib import Path class TraceRecorder: def __init__(self, path: str): self.path = Path(path) def record(self, event: DecisionEvent) -> None: with self.path.open("a", encoding="utf-8") as f: f.write(json.dumps(asdict(event), sort_keys=True) + "\n") This is enough to start.
You can later replace JSONL with:
Postgres, ClickHouse, OpenTelemetry-compatible storage, an event bus, a specialized tracing backend. The storage technology is secondary.
The semantic event model is the important part.
Add an explicit outcome event A run should end with something like:
@dataclass class RunOutcome: run_id: str state_id: str result: str failure_stage: str | None verified: bool verifier_result: str total_cost: float total_latency_ms: int This creates the bridge between trajectory events and benchmark outcomes.
Now you can aggregate production traces by:
architecture version model version router version failure stage task family cost band latency band verifier result Architecture versions belong in every trace Advanced agents change frequently.
A useful trace records:
architecture_version router_version prompt_version model_version scorer_version verifier_version memory_policy_version Otherwise a production dashboard may combine incompatible runs.
You cannot reliably compare a new router against an old one if version identity is missing.
Prompt versions are operational metadata You do not necessarily need to inline the full prompt into every event.
But record an immutable prompt identifier or hash.
For example:
prompt_id = planner:v17 prompt_sha = 5e84... That lets you later ask:
Did failures start after planner prompt v17?
The same principle applies to tool schemas and policy configurations.
Observability should support differential comparison A powerful production workflow is comparing two runs of the same task.
For example:
Run A: PASS, $0.08 Run B: FAIL, $0.17 A differential trace can show:
same router same planner same first two nodes then: A kept node C B pruned node C B expanded D twice B escalated to frontier model B critic rejected correct patch This turns debugging from document comparison into trajectory comparison.
Trace invariants You can validate the trace itself.
Useful invariants include:
every child node has a known parent selected node was previously generated pruned node is never later expanded without explicit restore event verification state ID exists every escalation has source and destination policies costs are non-negative run total equals component totals within tolerance final outcome references final state These checks catch observability corruption.
That matters because incorrect traces can lead you to incorrect conclusions.
Observability can become part of testing Once event structure is explicit, tests can assert architectural behavior.
For example:
def test_easy_task_does_not_escalate(trace): assert not any(e.kind == "ESCALATION_TRIGGERED" for e in trace) def test_failed_verification_triggers_recovery(trace): failed = [e for e in trace if e.kind == "VERIFICATION_RUN" and e.decision == "FAIL"] recovered = [e for e in trace if e.kind == "RECOVERY_TRIGGERED"] assert failed assert recovered This is a major step forward.
You are no longer only testing final outputs.
You are testing the orchestration policy.
Detect pathological loops from traces Trajectory observability makes repeated behavior easy to detect.
Fingerprint actions:
import hashlib import json def fingerprint(kind: str, payload: dict) -> str: raw = json.dumps({"kind": kind, "payload": payload}, sort_keys=True) return hashlib.sha256(raw.encode()).hexdigest() Then measure:
repeated action rate repeated state rate repeated critique rate repeated route rate A high repetition rate can reveal loops that token budgets only hide.
Detect duplicate search branches Search systems often spend compute on semantically identical branches.
Trace both:
node identity state fingerprint Then calculate:
unique state ratio = unique state fingerprints / generated nodes A low ratio indicates wasted exploration.
This connects observability directly to the beam-search and MCTS benchmarking metrics from the previous post.
Detect critic oscillation Suppose an agent alternates:
A → B → A → B because two critics disagree.
Trace candidate fingerprints and critic IDs.
Now detect cycles:
candidate_1 hash = X candidate_2 hash = Y candidate_3 hash = X candidate_4 hash = Y That is not productive revision.
It is oscillation.
The runtime can stop or escalate rather than spending indefinitely.
Observability enables adaptive agents safely Adaptive policies should learn from measurable signals.
Trajectory traces can provide features such as:
task family router confidence search branching factor verifier UNKNOWN rate critic correction rate historical escalation value cost of successful trajectories But adaptation should consume verified outcomes, not merely internal scores.
Otherwise the system can optimize toward its own mistaken beliefs.
This is where observability connects to learning from previous runs.
What not to optimize Do not blindly optimize:
critic approval router confidence agent confidence average model score number of completed steps These are internal signals.
They may correlate with success.
They are not success.
Prefer:
verified task success false-success rate cost per verified success latency to verified success recovery rate Internal signals are useful as explanatory features, not ground truth.
Production dashboards that actually help Once traces are trustworthy, useful dashboards include:
Outcome dashboard verified PASS FAIL UNKNOWN false-success rate Cost dashboard cost per verified success cost by component p50/p95 cost Routing dashboard route distribution routing accuracy missed escalation unnecessary escalation Search dashboard nodes generated nodes expanded pruning regret duplicate ratio branch diversity Critic dashboard wrong → correct correct → wrong accepted critiques rejected critiques Verification dashboard coverage UNKNOWN rate false-pass rate verifier latency Notice that these dashboards correspond directly to architectural mechanisms.
Alerts should target behavior, not only outages Traditional systems alert on:
CPU memory errors latency Advanced agents may also need alerts for:
false-success spike UNKNOWN spike routing drift cost-per-success spike pruning-regret spike duplicate-branch spike critic regression unnecessary escalation spike memory-staleness spike These signals reveal degradation before the service necessarily crashes.
Sampling traces Full trajectory traces can be expensive.
You may not need identical retention for every run.
One strategy:
100% failures 100% UNKNOWN 100% high-cost outliers 100% safety-sensitive actions sample successful routine runs Keep aggregated metrics for all runs while retaining detailed payloads selectively.
This balances cost with debuggability.
Do not sample away rare failures Random sampling can miss precisely the events you care about.
Prefer targeted retention for:
unexpected escalation verifier disagreement correct → wrong critique transitions high pruning regret rollback events irreversible side effects policy violations Rare bad trajectories are often more valuable than common successful ones.
From debugging to science There is a deeper consequence of structured trajectory observability.
Your production system begins generating a dataset of:
state choice alternatives evidence cost outcome That can support:
router calibration, scorer improvement, search-policy evaluation, critic evaluation, adaptive budgeting, memory promotion, failure clustering, benchmark creation. But only if the outcome evidence is trustworthy.
Otherwise you train future policy on noisy self-assessment.
A closed learning loop A mature system can eventually operate like this:
production trajectories ↓ verified outcomes ↓ failure classification ↓ offline replay ↓ architecture experiment ↓ compute-matched benchmark ↓ new policy version ↓ production canary ↓ new trajectories This is far more disciplined than continuously adding new agent mechanisms.
The architecture evolves from evidence.
A practical debugging workflow When an advanced agent fails, investigate in this order.
1. Verify the failure Confirm that the final state actually failed the external acceptance criteria.
2. Identify the final state Make sure the verifier tested the same state the agent returned.
3. Classify the failure stage Was it generation, selection, routing, execution, memory, verification or recovery?
4. Inspect the decisive event Which decision most directly changed the trajectory toward failure?
5. Inspect alternatives Was a better route or branch available?
6. Inspect evidence Was the decision reasonable given the evidence available at the time?
7. Inspect cost Did the system spend significant compute without improving the trajectory?
8. Replay counterfactually Change one decision and test whether the outcome improves.
9. Aggregate Determine whether this is an isolated case or a systematic architectural failure.
That is trajectory debugging.
The deepest rule The more advanced your agent becomes, the easier it is to hide mistakes inside orchestration.
A wrong final answer may be caused by:
bad generation bad selection bad routing bad pruning bad critique bad memory bad verification bad recovery bad budget allocation If your logs collapse all of those into:
agent failed then you have built a system you cannot meaningfully improve.
The goal of observability is not to collect more text.
It is to preserve the causal structure of the run.
That gives us the central rule for this stage of advanced-agent engineering:
If an architecture makes a decision that can change the outcome, that decision should leave a trace containing the state, alternatives, evidence, reason, cost and consequence.
Once you have that, advanced agents stop being mysterious bundles of prompts.
They become inspectable computational systems.
And that sets up the next stage naturally:
How do you use those production trajectories to improve the architecture without teaching the system from its own mistakes?