An advanced agent fails.

You look at the final answer.

It is wrong.

So you inspect the prompt.

The prompt looks reasonable.

You inspect the model response.

That also looks reasonable.

But somewhere between the original request and the final result the system:

• chose the wrong specialist,
• pruned the branch that contained the right solution,
• trusted a critic that was wrong,
• escalated to an expensive model unnecessarily,
• failed to escalate when it should have,
• retrieved stale memory,
• spent most of its budget exploring duplicates,
• accepted a weak verifier signal,
• retried the same strategy under a different name,
• or transformed a local success into a global failure.

The final answer does not tell you which one happened.

Neither does a token count.

Neither does a single log line saying:

agent completed successfully

That is the production problem this post solves.

Advanced agents need observability at the level where decisions are actually made.

If the architecture contains routers, planners, critics, search trees, specialists, memory, adaptive budgets and external verification, then the trace must expose those mechanisms explicitly.

Otherwise your system may be sophisticated at runtime and primitive at debugging time.

The first-principles problem

A simple application trace often looks like this:

request
  ↓
function call
  ↓
database
  ↓
response

An advanced agent may instead look like this:

request
  ↓
classify task
  ↓
route to specialist
  ↓
generate candidate states
  ↓
score branches
  ↓
prune branches
  ↓
escalate one branch
  ↓
critic review
  ↓
revision
  ↓
tool calls
  ↓
verify result
  ↓
final answer

A conventional trace that only records model calls will miss the most important events.

The agent’s behavior is not merely a sequence of prompts.

It is a sequence of decisions about computation.

Those decisions need lineage.

Observability is not logging everything

There is an easy but dangerous response to this problem:

Log everything.

That gives you enormous traces containing:

• prompts,
• completions,
• embeddings,
• tool payloads,
• every branch,
• every token,
• every retrieved memory,
• every intermediate score.

Then a production failure occurs and nobody can determine what matters.

High-volume logging is not the same thing as observability.

Useful agent observability answers specific questions:

What decision happened?
Why did it happen?
What alternatives existed?
What evidence was available then?
What state did the decision operate on?
How much did it cost?
What happened because of it?
Was the final outcome verified?

The unit of observability should therefore be the decision event.

A decision event

A useful generic event can be represented like this:

from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Any


@dataclass
class DecisionEvent:
    event_id: str
    run_id: str
    parent_event_id: str | None
    kind: str
    component: str
    state_id: str
    timestamp: str
    decision: str
    reason: str
    alternatives: list[str] = field(default_factory=list)
    evidence_ids: list[str] = field(default_factory=list)
    model: str | None = None
    prompt_version: str | None = None
    cost: float = 0.0
    latency_ms: int = 0
    metadata: dict[str, Any] = field(default_factory=dict)


def now_iso() -> str:
    return datetime.now(timezone.utc).isoformat()

The exact schema is not important.

The separation is.

A decision event should distinguish:

state
choice
reason
alternatives
evidence
cost
outcome

That is enough to reconstruct much of the causal path later.

Run IDs are not enough

Most systems already have a trace ID or run ID.

That identifies one execution.

But an advanced agent contains branching structure.

You need identities at several levels:

run_id
  ↓
trajectory_id
  ↓
state_id
  ↓
node_id
  ↓
event_id
  ↓
evidence_id

These identities solve different problems.

A run_id answers:

Which user request did this belong to?

A trajectory_id answers:

Which candidate path did this action belong to?

A state_id answers:

What exact world or repository state was being considered?

A node_id answers:

Which search node did we expand, evaluate or prune?

An evidence_id answers:

What observation justified the decision?

Without these distinctions, branches from the same run become difficult to reconstruct.

State identity matters

Suppose a coding agent runs tests on commit A.

It then modifies the repository to commit B.

The tests from A passed.

If your trace only says:

tests: PASS

then the evidence can accidentally be associated with B.

A stronger trace says:

verification_event:
    state_id: git-tree:7f91...
    verifier: pytest
    result: PASS

Now the evidence is bound to the exact state it tested.

This is the same principle we used in the verification post:

Evidence without state identity can become stale evidence.

Observability must preserve that relationship.

The trace is a graph, not a list

Sequential logs work well when execution is sequential.

Search is not sequential.

Consider a Tree of Thoughts run:

              root
           /    |    \
          A     B     C
         / \          \
       A1  A2          C1
            \
             A2a

A timestamp-ordered log might show:

create A
create B
create C
score A
score B
score C
create A1
create A2
create C1
score A1
...

That tells you execution order.

It does not clearly tell you lineage.

Store the graph explicitly:

@dataclass
class SearchNodeTrace:
    node_id: str
    parent_id: str | None
    trajectory_id: str
    state_id: str
    depth: int
    action: str
    score: float | None
    status: str
    reason: str | None

Possible statuses might include:

GENERATED
EVALUATED
EXPANDED
PRUNED
SELECTED
VERIFIED
FAILED

Now you can ask a much more useful debugging question:

Which ancestor decision led to the final failure?

Record why a branch was pruned

A search trace that says:

node B7 -> PRUNED

is incomplete.

Why was it pruned?

Possible reasons include:

score below beam threshold
hard constraint violated
duplicate state
budget exhausted
verifier failed
risk policy rejected
parent dominated
stale state

Record the reason as structured data.

For example:

@dataclass
class PruneEvent:
    node_id: str
    state_id: str
    reason_code: str
    score: float | None
    threshold: float | None
    competing_node_ids: list[str]

Then you can measure pruning behavior across thousands of tasks.

That matters because the benchmark post introduced pruning regret.

Observability gives you the data needed to calculate it in production.

Pruning regret needs lineage

Suppose branch C was pruned early.

Later offline replay shows that C would have produced a verified solution.

You now know the search system committed a selection error.

Conceptually:

pruned branch
     ↓
offline continuation
     ↓
verified success
     ↓
pruning regret

Without branch identity and prune reasons, this analysis becomes guesswork.

With them, you can ask:

Which scorer pruned the successful branch?
At what depth?
Against which competing branch?
At what score margin?
How often does this happen?

That is a production debugging loop.

Routing decisions need the same treatment

Mixture-of-experts and mixture-of-agents systems introduce another hidden decision:

task
  ↓
router
  ↓
expert

If the expert fails, the router may be the real problem.

So route traces should capture:

@dataclass
class RouteEvent:
    task_id: str
    state_id: str
    available_routes: list[str]
    selected_route: str
    scores: dict[str, float]
    reason: str
    escalation_allowed: bool

This lets you distinguish:

expert failure
vs
routing failure

That distinction is crucial.

If a coding specialist fails on a legal-research task, improving the coding specialist is irrelevant.

The route was wrong.

Routing regret

The benchmarking post introduced cost-weighted routing regret.

Observability lets you compute it from actual production traces.

Suppose the router chose:

frontier_model
cost = $0.40
verified = PASS

But replay shows:

local_specialist
cost = $0.01
verified = PASS

The original route succeeded.

It was still wasteful.

Conversely:

local_model
cost = $0.01
verified = FAIL

when a specialist would have passed is a missed escalation.

The trace must preserve enough information to tell those apart.

Escalation should be observable

Adaptive agents often use escalation policies:

cheap path
   ↓
uncertainty high?
   ├─ no -> continue
   └─ yes -> stronger model

A production trace should explain:

why escalation happened
what threshold fired
what extra cost was incurred
whether escalation changed the outcome

For example:

@dataclass
class EscalationEvent:
    from_policy: str
    to_policy: str
    signal_name: str
    signal_value: float
    threshold: float
    incremental_cost: float

Then you can measure:

useful escalation rate
unnecessary escalation rate
missed escalation rate
cost per recovered failure

Without these events, adaptive systems become difficult to calibrate.

Critics need attribution

Planner/executor/critic systems can fail in several ways.

The generator may be wrong.

The critic may correctly identify the error.

The reviser may ignore the criticism.

Or the critic may damage a correct answer.

A final output does not tell you which happened.

Trace the transitions:

candidate_v1
   ↓
critic_1
   ↓
critique
   ↓
candidate_v2
   ↓
external verifier

Record:

before score
critic finding
revision applied?
after score
external outcome

Then calculate:

wrong → correct
correct → correct
wrong → wrong
correct → wrong

The last category is especially important.

A critic that frequently turns correct solutions into failures is not helping merely because its prose sounds intelligent.

Multi-agent debate needs disagreement traces

A debate system adds even more intermediate structure.

For every proposition or candidate, you may have:

agent A position
agent B position
agent C position
judge decision
final revision
external verification

Useful observability should preserve:

initial disagreement
independence of positions
arguments introduced
position changes
judge selection
final correctness

Otherwise you cannot distinguish genuine error correction from social convergence.

Three agents agreeing after reading the same mistaken evidence is not diversity.

It is correlated failure.

Do not confuse hidden reasoning with operational reasoning

An observability system should not depend on storing private internal model reasoning.

You do not need hidden chain-of-thought to debug the runtime.

You need structured operational decisions:

selected tool X
because schema matched intent Y

pruned node B
because score 0.31 < beam cutoff 0.54

escalated model
because verifier returned UNKNOWN twice

selected candidate C
because test coverage passed and latency was lower

These are runtime facts.

They are much more useful for engineering than an enormous free-form reasoning transcript.

A strong design records decision summaries and evidence, not dependence on hidden model internals.

Cost is part of the trajectory

Advanced agents allocate computation dynamically.

Therefore every meaningful event should expose cost.

At minimum:

input tokens
output tokens
model calls
tool calls
latency
monetary cost

Then aggregate by component:

router        $0.002
planner       $0.020
search        $0.110
critic        $0.025
verifier      $0.004
---------------
total         $0.161

Now debugging changes.

You can ask:

Why did this request cost ten times more than the median?

Perhaps:

• beam search produced duplicates,
• MCTS over-expanded one subtree,
• the critic oscillated,
• the router repeatedly escalated,
• verification returned UNKNOWN and triggered retries.

Cost attribution makes those mechanisms visible.

Useful cost metrics

Per run, record:

total_cost
model_cost
tool_cost
verification_cost
search_cost
recovery_cost

Across runs, calculate:

cost per verified success
cost per recovered failure
cost per search node
cost per useful escalation
cost per accepted critique

These are more informative than total token usage alone.

Latency needs critical-path attribution

Parallel agents complicate latency.

Suppose three specialists run concurrently:

A: 800 ms
B: 1200 ms
C: 4500 ms

The total work is 6500 ms.

The wall-clock contribution is closer to 4500 ms.

Observability should distinguish:

work time
vs
critical-path latency

This matters for tree search, debate, ensemble generation and specialist fan-out.

A component can consume significant compute without increasing user-visible latency if it runs in parallel.

Or one slow branch can dominate the entire request.

Verification evidence belongs in the trace

The final result should not simply say:

success = true

Record the verifier evidence.

For example:

@dataclass
class VerificationTrace:
    verifier_id: str
    verifier_version: str
    state_id: str
    result: str  # PASS | FAIL | UNKNOWN
    criteria: dict[str, str]
    evidence_ids: list[str]
    latency_ms: int
    cost: float

This lets you answer:

Which criterion failed?
Which verifier version produced the result?
Was the evidence current?
Was the verifier deterministic?
Did the agent continue after UNKNOWN?

Verification is not just the last line of the trace.

It is part of the causal graph.

UNKNOWN is observable information

A sophisticated agent should sometimes terminate with:

UNKNOWN

That can happen when:

• required evidence is unavailable,
• an external system is unreachable,
• verification coverage is incomplete,
• a deployment state changed during checking,
• the acceptance test is inconclusive.

Trace the reason.

A large UNKNOWN rate might indicate:

weak verifier coverage
bad environment access
poor evidence collection
race conditions
insufficient task specification

If UNKNOWN gets collapsed into FAIL or PASS, you lose that diagnostic signal.

Memory needs read and write lineage

Memory introduces two important events:

memory read
memory write

A memory read should record:

query
scope
candidate IDs
selected IDs
scores
freshness
provenance

A memory write should record:

source event
verification status
memory type
scope
TTL/version
promotion reason

Then when an agent follows stale information, you can determine whether the problem was:

bad stored memory
bad retrieval
bad ranking
bad freshness filtering
bad context assembly

Without memory lineage, these failures blur together.

Tool calls need before-and-after state

A tool invocation trace should not only capture arguments and output.

For side-effecting tools, record:

precondition state
requested action
execution result
postcondition state
verification result

Conceptually:

before_state
    ↓
tool action
    ↓
reported result
    ↓
after_state
    ↓
verification

This reveals a common agent failure:

tool returned 200 OK

but the intended user outcome did not occur.

HTTP success is not necessarily goal success.

A minimal event taxonomy

You do not need hundreds of event types.

A useful first version might have:

RUN_STARTED
STATE_OBSERVED
ROUTE_SELECTED
PLAN_CREATED
NODE_GENERATED
NODE_SCORED
NODE_PRUNED
NODE_SELECTED
MEMORY_READ
MEMORY_WRITTEN
TOOL_CALLED
TOOL_RESULT
CRITIQUE_CREATED
REVISION_CREATED
ESCALATION_TRIGGERED
VERIFICATION_RUN
RECOVERY_TRIGGERED
RUN_COMPLETED

The exact names do not matter.

Consistency does.

Separate events from payloads

Large prompts, model responses, files and tool outputs can make event streams enormous.

Store references instead:

event
  ↓
payload_id
  ↓
object/blob store

For example:

@dataclass
class PayloadRef:
    payload_id: str
    sha256: str
    media_type: str
    size_bytes: int
    location: str

This gives you compact traces while preserving reproducibility.

Hashing also helps detect whether two events operated on the same underlying content.

Redaction and privacy are part of observability design

Agent traces may contain:

• source code,
• customer data,
• credentials,
• proprietary documents,
• browser sessions,
• support tickets,
• internal policies.

Do not treat traces as harmless debug text.

Design for:

redaction
field-level access
retention limits
tenant isolation
secret filtering
deletability

An observability system that leaks the information it observes is not production-ready.

Replay is the next level

A trace helps explain what happened.

Replay helps test what would happen differently.

There are several forms.

Exact replay

Reuse recorded outputs and deterministic state transitions.

Useful for debugging orchestration logic.

Model replay

Rerun the same prompts against the same model version where possible.

Useful for measuring stochastic variation.

Counterfactual replay

Change one decision:

use another route
keep pruned node
skip critic
change beam width
avoid escalation

and continue the trajectory.

This is enormously useful for advanced agents.

It turns production traces into architecture experiments.

Counterfactual debugging

Suppose a failed coding run used MCTS.

The selected path was:

root → B → B2 → B2a

But branch A was pruned.

Counterfactual replay can ask:

What if A had survived one more expansion?

If A then reaches verified success, the problem is likely the search policy or evaluator.

If A also fails, pruning may not be the issue.

This is much stronger than staring at the final patch.

Replay must control side effects

Never casually replay production actions against production systems.

Prefer:

sandbox
fixture
mock service
read-only mode
transaction rollback
container/worktree
staging environment
recorded tool output

The replay engine should know which events are:

pure
read-only
reversible
irreversible

Counterfactual search over irreversible actions without isolation is a dangerous architecture.

Failure localization

Once traces are structured, you can classify failures by stage.

For example:

GENERATION_FAILURE
SELECTION_FAILURE
ROUTING_FAILURE
PLANNING_FAILURE
EXECUTION_FAILURE
MEMORY_FAILURE
VERIFICATION_FAILURE
RECOVERY_FAILURE
BUDGET_FAILURE

A failed run can have more than one label.

But even approximate localization changes engineering priorities.

If 60% of failures are routing failures, tuning the generator is probably the wrong investment.

If most failures are verification UNKNOWNs, improve evidence access.

If search frequently generates a successful branch but prunes it, improve selection.

Observability gives you the evidence to know which layer is failing.

Failure decomposition with oracle@N

Suppose a search system generates ten candidates.

One would pass the external verifier.

But the selected candidate fails.

Then:

oracle@10 = success
selected result = failure

That is primarily a selection failure.

If none of the ten candidates would pass, it is primarily a generation failure.

This diagnostic only works if candidate identities and outcomes are traceable.

Failure decomposition with critics

Suppose:

candidate A = correct
critic says A is wrong
revision B = incorrect

The final run fails.

Without attribution you might blame generation.

With trajectory observability you identify:

correct → wrong
caused by critic/revision stage

That is an entirely different engineering action.

Failure decomposition with routers

Suppose a task fails using general_model.

Offline evaluation shows code_specialist succeeds.

That is a routing miss.

Suppose the router selected code_specialist correctly but it still failed.

That is an expert capability failure.

Again, the final output alone cannot distinguish them.

Search-tree visualizations are useful, but secondary

A graphical tree can be very helpful:

root
├── A score=.61
│   ├── A1 score=.69 SELECTED
│   └── A2 score=.23 PRUNED
├── B score=.59 PRUNED
└── C score=.18 PRUNED

But the underlying structured events matter more than the visualization.

If your data model is correct, you can build:

• tree views,
• timelines,
• flame graphs,
• routing matrices,
• cost charts,
• verifier dashboards,
• replay tools.

If your data model is weak, dashboards merely make weak data prettier.

A useful run summary

After a run, generate a concise structured summary:

Run: 8a31
Outcome: FAIL
State: git-tree:4d2a

Route:
  coding-specialist

Search:
  generated: 18 nodes
  expanded: 7
  pruned: 11
  duplicate ratio: 0.28

Critique:
  2 critiques
  1 revision accepted

Verification:
  tests: FAIL
  lint: PASS

Cost:
  total: $0.14
  search: $0.09
  critic: $0.03
  verification: $0.02

Likely failure stage:
  SELECTION_FAILURE

That is far more useful than dumping every prompt first.

Observability for coding agents

Coding agents have unusually strong state identities and verifiers.

Useful state IDs include:

git commit
git tree hash
worktree ID
patch hash
test environment hash

Important events include:

files inspected
symbols selected
plan created
patch generated
tests run
branch chosen
critic findings
verification state

Useful debugging questions:

Why did the agent edit this file?
Which evidence connected it to the task?
Which alternative patch was rejected?
Which test failure caused replanning?
Did the final tests run against the final tree?

This is where decision lineage becomes extremely powerful.

Observability for research agents

Research agents need strong source provenance.

Track:

query
source IDs
publication dates
claim IDs
support relationships
contradictions
source quality
retrieval rank

A useful graph is:

claim
  ↓
source evidence
  ↓
extraction
  ↓
synthesis
  ↓
final statement

When a claim is wrong, you can determine whether:

• retrieval missed the right source,
• extraction misread it,
• synthesis overgeneralized,
• stale information outranked current evidence,
• verification coverage was missing.

Observability for browser agents

Browser agents require state snapshots around navigation and mutation.

Track:

URL/page identity
DOM/state hash
form state
selected action
tool result
post-action page state

A common failure is:

click reported success

but the page did not transition as expected.

The trace should show both.

For destructive actions such as purchases, submissions or deletions, provenance and pre/post-condition checks become especially important.

Observability for data agents

Data agents should bind actions to:

dataset version
schema version
pipeline run ID
query hash
transformation hash
validation result

Then a bad transformation can be traced through:

source version
  ↓
selected operation
  ↓
transformed state
  ↓
validation

This also helps reproduce failures after underlying datasets change.

Observability for DevOps agents

DevOps agents operate in consequential environments.

Track:

cluster/deployment state
incident ID
change request
precondition checks
selected remediation
approval state
execution result
postcondition verification
rollback availability

For example:

restart service

should not be represented as a lone action event.

You want:

why restart was selected
what evidence supported it
what alternatives existed
whether approval was required
what state changed
whether health recovered
whether rollback was needed

That is operationally meaningful observability.

Observability for mixture-of-agents systems

A mixture-of-agents runtime may combine:

router
retriever
planner
local specialist
frontier specialist
search
critic
verifier

The trace should allow a waterfall like:

request
  ↓
router: code task (.92)
  ↓
local coder
  ↓
verifier: UNKNOWN
  ↓
escalation
  ↓
frontier coder
  ↓
critic
  ↓
revision
  ↓
verifier: PASS

Then cost attribution can explain exactly where the extra compute was spent.

This is essential for optimizing heterogeneous systems.

Build traces before dashboards

A practical implementation order is:

1. stable IDs
2. event schema
3. state identity
4. evidence references
5. cost attribution
6. failure classification
7. replay
8. dashboards

Do not begin with a beautiful UI.

Begin with data you can trust.

A tiny trace recorder

A minimal implementation can be extremely small:

import json
from dataclasses import asdict
from pathlib import Path


class TraceRecorder:
    def __init__(self, path: str):
        self.path = Path(path)

    def record(self, event: DecisionEvent) -> None:
        with self.path.open("a", encoding="utf-8") as f:
            f.write(json.dumps(asdict(event), sort_keys=True) + "\n")

This is enough to start.

You can later replace JSONL with:

• Postgres,
• ClickHouse,
• OpenTelemetry-compatible storage,
• an event bus,
• a specialized tracing backend.

The storage technology is secondary.

The semantic event model is the important part.

Add an explicit outcome event

A run should end with something like:

@dataclass
class RunOutcome:
    run_id: str
    state_id: str
    result: str
    failure_stage: str | None
    verified: bool
    verifier_result: str
    total_cost: float
    total_latency_ms: int

This creates the bridge between trajectory events and benchmark outcomes.

Now you can aggregate production traces by:

architecture version
model version
router version
failure stage
task family
cost band
latency band
verifier result

Architecture versions belong in every trace

Advanced agents change frequently.

A useful trace records:

architecture_version
router_version
prompt_version
model_version
scorer_version
verifier_version
memory_policy_version

Otherwise a production dashboard may combine incompatible runs.

You cannot reliably compare a new router against an old one if version identity is missing.

Prompt versions are operational metadata

You do not necessarily need to inline the full prompt into every event.

But record an immutable prompt identifier or hash.

For example:

prompt_id = planner:v17
prompt_sha = 5e84...

That lets you later ask:

Did failures start after planner prompt v17?

The same principle applies to tool schemas and policy configurations.

Observability should support differential comparison

A powerful production workflow is comparing two runs of the same task.

For example:

Run A: PASS, $0.08
Run B: FAIL, $0.17

A differential trace can show:

same router
same planner
same first two nodes

then:
A kept node C
B pruned node C

B expanded D twice
B escalated to frontier model
B critic rejected correct patch

This turns debugging from document comparison into trajectory comparison.

Trace invariants

You can validate the trace itself.

Useful invariants include:

every child node has a known parent
selected node was previously generated
pruned node is never later expanded without explicit restore event
verification state ID exists
every escalation has source and destination policies
costs are non-negative
run total equals component totals within tolerance
final outcome references final state

These checks catch observability corruption.

That matters because incorrect traces can lead you to incorrect conclusions.

Observability can become part of testing

Once event structure is explicit, tests can assert architectural behavior.

For example:

def test_easy_task_does_not_escalate(trace):
    assert not any(e.kind == "ESCALATION_TRIGGERED" for e in trace)


def test_failed_verification_triggers_recovery(trace):
    failed = [e for e in trace if e.kind == "VERIFICATION_RUN" and e.decision == "FAIL"]
    recovered = [e for e in trace if e.kind == "RECOVERY_TRIGGERED"]
    assert failed
    assert recovered

This is a major step forward.

You are no longer only testing final outputs.

You are testing the orchestration policy.

Detect pathological loops from traces

Trajectory observability makes repeated behavior easy to detect.

Fingerprint actions:

import hashlib
import json


def fingerprint(kind: str, payload: dict) -> str:
    raw = json.dumps({"kind": kind, "payload": payload}, sort_keys=True)
    return hashlib.sha256(raw.encode()).hexdigest()

Then measure:

repeated action rate
repeated state rate
repeated critique rate
repeated route rate

A high repetition rate can reveal loops that token budgets only hide.

Detect duplicate search branches

Search systems often spend compute on semantically identical branches.

Trace both:

node identity
state fingerprint

Then calculate:

unique state ratio = unique state fingerprints / generated nodes

A low ratio indicates wasted exploration.

This connects observability directly to the beam-search and MCTS benchmarking metrics from the previous post.

Detect critic oscillation

Suppose an agent alternates:

A → B → A → B

because two critics disagree.

Trace candidate fingerprints and critic IDs.

Now detect cycles:

candidate_1 hash = X
candidate_2 hash = Y
candidate_3 hash = X
candidate_4 hash = Y

That is not productive revision.

It is oscillation.

The runtime can stop or escalate rather than spending indefinitely.

Observability enables adaptive agents safely

Adaptive policies should learn from measurable signals.

Trajectory traces can provide features such as:

task family
router confidence
search branching factor
verifier UNKNOWN rate
critic correction rate
historical escalation value
cost of successful trajectories

But adaptation should consume verified outcomes, not merely internal scores.

Otherwise the system can optimize toward its own mistaken beliefs.

This is where observability connects to learning from previous runs.

What not to optimize

Do not blindly optimize:

critic approval
router confidence
agent confidence
average model score
number of completed steps

These are internal signals.

They may correlate with success.

They are not success.

Prefer:

verified task success
false-success rate
cost per verified success
latency to verified success
recovery rate

Internal signals are useful as explanatory features, not ground truth.

Production dashboards that actually help

Once traces are trustworthy, useful dashboards include:

Outcome dashboard

verified PASS
FAIL
UNKNOWN
false-success rate

Cost dashboard

cost per verified success
cost by component
p50/p95 cost

Routing dashboard

route distribution
routing accuracy
missed escalation
unnecessary escalation

Search dashboard

nodes generated
nodes expanded
pruning regret
duplicate ratio
branch diversity

Critic dashboard

wrong → correct
correct → wrong
accepted critiques
rejected critiques

Verification dashboard

coverage
UNKNOWN rate
false-pass rate
verifier latency

Notice that these dashboards correspond directly to architectural mechanisms.

Alerts should target behavior, not only outages

Traditional systems alert on:

CPU
memory
errors
latency

Advanced agents may also need alerts for:

false-success spike
UNKNOWN spike
routing drift
cost-per-success spike
pruning-regret spike
duplicate-branch spike
critic regression
unnecessary escalation spike
memory-staleness spike

These signals reveal degradation before the service necessarily crashes.

Sampling traces

Full trajectory traces can be expensive.

You may not need identical retention for every run.

One strategy:

100% failures
100% UNKNOWN
100% high-cost outliers
100% safety-sensitive actions
sample successful routine runs

Keep aggregated metrics for all runs while retaining detailed payloads selectively.

This balances cost with debuggability.

Do not sample away rare failures

Random sampling can miss precisely the events you care about.

Prefer targeted retention for:

unexpected escalation
verifier disagreement
correct → wrong critique transitions
high pruning regret
rollback events
irreversible side effects
policy violations

Rare bad trajectories are often more valuable than common successful ones.

From debugging to science

There is a deeper consequence of structured trajectory observability.

Your production system begins generating a dataset of:

state
choice
alternatives
evidence
cost
outcome

That can support:

• router calibration,
• scorer improvement,
• search-policy evaluation,
• critic evaluation,
• adaptive budgeting,
• memory promotion,
• failure clustering,
• benchmark creation.

But only if the outcome evidence is trustworthy.

Otherwise you train future policy on noisy self-assessment.

A closed learning loop

A mature system can eventually operate like this:

production trajectories
        ↓
verified outcomes
        ↓
failure classification
        ↓
offline replay
        ↓
architecture experiment
        ↓
compute-matched benchmark
        ↓
new policy version
        ↓
production canary
        ↓
new trajectories

This is far more disciplined than continuously adding new agent mechanisms.

The architecture evolves from evidence.

A practical debugging workflow

When an advanced agent fails, investigate in this order.

1. Verify the failure

Confirm that the final state actually failed the external acceptance criteria.

2. Identify the final state

Make sure the verifier tested the same state the agent returned.

3. Classify the failure stage

Was it generation, selection, routing, execution, memory, verification or recovery?

4. Inspect the decisive event

Which decision most directly changed the trajectory toward failure?

5. Inspect alternatives

Was a better route or branch available?

6. Inspect evidence

Was the decision reasonable given the evidence available at the time?

7. Inspect cost

Did the system spend significant compute without improving the trajectory?

8. Replay counterfactually

Change one decision and test whether the outcome improves.

9. Aggregate

Determine whether this is an isolated case or a systematic architectural failure.

That is trajectory debugging.

The deepest rule

The more advanced your agent becomes, the easier it is to hide mistakes inside orchestration.

A wrong final answer may be caused by:

bad generation
bad selection
bad routing
bad pruning
bad critique
bad memory
bad verification
bad recovery
bad budget allocation

If your logs collapse all of those into:

agent failed

then you have built a system you cannot meaningfully improve.

The goal of observability is not to collect more text.

It is to preserve the causal structure of the run.

That gives us the central rule for this stage of advanced-agent engineering:

If an architecture makes a decision that can change the outcome, that decision should leave a trace containing the state, alternatives, evidence, reason, cost and consequence.

Once you have that, advanced agents stop being mysterious bundles of prompts.

They become inspectable computational systems.

And that sets up the next stage naturally:

How do you use those production trajectories to improve the architecture without teaching the system from its own mistakes?