# Advanced Agents From First Principles 26: Why Did the Agent Fail? Build an Incident Forensics Pipeline
# Book: Advanced Agents From First Principles
# Chapter URL: https://programmer.ie/books/advanced-agents-from-first-principles/26-chapter/
# Prose fingerprint (sha256, normalized): 2d7ac2639e200df89b7f6bed1de5c516c8ba4dd5bb828c68751f3501ed0ef0e8
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

An agent can fail even when every service is up.
The model responded.
The router returned a route.
The browser returned HTML.
The tool call returned 200.
The verifier returned PASS.
The run completed.
And the result was still wrong.
That is the uncomfortable point at which ordinary infrastructure debugging stops being enough.
If you run advanced agents in production, sooner or later you need to answer a harder question:
Why did this specific run fail, where did the failure first become inevitable, and would the proposed fix actually have prevented it?
That is an incident-forensics problem.
Step 25 gave us deterministic replay and provenance. We can reconstruct the behavioral release, prompts, observations, tool outputs, environment snapshots, policy decisions, branch lineage, side effects, and verifier evidence associated with a historical run.
That gives us the evidence.
It does not automatically give us the cause.
A useful incident-forensics pipeline must do more than produce a timeline.
It must distinguish:
the first wrong observation from the first wrong decision, the first wrong decision from later symptoms, model error from routing error, stale state from tool failure, candidate failure from selection failure, verifier failure from execution failure, local failure from systemic failure, correlation from causation, a plausible remediation from a remediation that actually prevents recurrence. The core rule for this post is:
Find the earliest evidence-backed divergence from a successful trajectory, not merely the last visible error.
That one principle changes how you investigate agent incidents.
The Search Problem: “AI Agent Wrong Answer Root Cause” A common debugging pattern looks like this:
bad final result ↓ inspect final model output ↓ blame the model That is usually too shallow.
The model may have produced the wrong final answer because:
the router sent the task to the wrong expert, retrieval supplied stale evidence, the environment snapshot was already outdated, a search branch containing the correct answer was pruned, a critic damaged a previously correct candidate, the browser tool silently landed on a fallback page, a retry duplicated a state-changing action, a stale distributed worker committed after losing ownership, the verifier checked the wrong artifact, the verifier itself drifted, the scheduler starved the run of verification budget, a fallback model did not support the required capability, a release migration changed semantics without changing the schema. The final model call may only be the last participant in a failure chain that began much earlier.
So the correct question is not:
Which component emitted the bad result?
It is:
At what point did the execution trajectory first diverge from the set of trajectories that could still satisfy the task contract?
That is the earliest divergence.
1. Define the Incident Before Explaining It Do not begin with a root-cause story.
Begin with an incident contract.
from dataclasses import dataclass from enum import Enum from typing import Optional class IncidentSeverity(str, Enum): LOW = "low" MEDIUM = "medium" HIGH = "high" CRITICAL = "critical" @dataclass(frozen=True) class IncidentRecord: incident_id: str run_id: str detected_at: str severity: IncidentSeverity expected_contract: str observed_outcome: str verifier_outcome: str external_impact: str initial_detection_source: str release_id: str tenant_id: Optional[str] = None Notice what is absent:
root_cause="the model hallucinated" That belongs later.
The incident record should capture what happened before anyone decides why it happened.
A clean incident statement might be:
Expected: The coding agent must modify only files under src/payments/, run the payment test suite, and return PASS only when the resulting worktree matches the requested behavior. Observed: The agent modified src/payments/ and src/auth/, the payment tests passed, and the verifier returned PASS because its scope check used the pre-edit file list rather than the post-edit worktree. That is already much more useful than:
The AI made an unsafe change. 2. Preserve the Evidence Before Reproducing Anything A production incident can disappear while you investigate it.
The model version changes.
The retrieval index refreshes.
The branch moves.
The browser session expires.
The database state advances.
A worker gets reassigned.
A prompt template gets deployed.
So the first forensic action is preservation.
incident detected ↓ freeze identifiers ↓ preserve replay manifest ↓ preserve observations ↓ preserve candidate artifacts ↓ preserve verifier evidence ↓ preserve side-effect records ↓ preserve environment snapshot references The Step 25 replay manifest becomes your incident evidence bundle.
At minimum preserve:
run_id release_id model version(s) prompt hashes router policy version search policy version budget policy version verifier version tool registry version retrieval snapshot memory snapshot boundary environment snapshot operation IDs attempt IDs lease/fencing epochs candidate hashes observation hashes verifier evidence hashes final outcome Do not start by rerunning the task against current production.
That is a new run.
3. Build a Causal Timeline, Not Just a Log Timeline Ordinary logs tell you what happened in timestamp order.
Forensics needs causal order.
Suppose this happened:
10:01:00 retrieval snapshot loaded 10:01:02 branch A generated 10:01:03 branch B generated 10:01:04 branch A scored 0.71 10:01:04 branch B scored 0.69 10:01:05 branch B pruned 10:01:06 branch A executed tool 10:01:08 verifier returned FAIL 10:01:09 critic revised A 10:01:11 verifier returned PASS A timestamp list does not answer:
Did the stale retrieval cause branch A to be wrong? Was branch B actually correct? Did the scorer prune the correct branch? Did the critic fix the candidate or merely satisfy a weak verifier? Was the PASS evidence attached to the final candidate? Represent dependencies explicitly.
@dataclass(frozen=True) class ForensicEdge: source_event_id: str target_event_id: str relation: str # Examples: # observation -> routing_decision # routing_decision -> candidate_generation # candidate -> score # score -> prune_decision # tool_result -> candidate_revision # candidate_hash -> verifier_evidence Then the incident is a graph:
observation O1 ↓ route R1 ↓ candidate C1 ─────→ score S1 ─────→ selected │ └──────────────→ tool T1 ↓ candidate C2 ↓ verifier V1 ↓ PASS This is much easier to investigate than a flat wall of logs.
4. Separate Root Cause From Trigger, Amplifier, and Detector Failure Many incidents have more than one important cause.
A useful taxonomy is:
trigger root cause amplifier containment failure detection failure Example:
Trigger: A provider changed HTML structure. Root cause: The browser parser accepted missing structured fields as an empty result. Amplifier: The retry policy retried the same malformed extraction five times. Containment failure: No circuit breaker opened for semantic parser failures. Detection failure: The verifier checked request completion, not extracted-field completeness. If you record only:
root cause = website changed you will probably fix the wrong layer.
The website changing may be inevitable.
The preventable part was that your system interpreted malformed output as valid state and failed to contain it.
5. The First Divergence Principle Imagine a coding-agent run.
Task ↓ route to code specialist ↓ retrieve repository context ↓ generate patch A ↓ generate patch B ↓ score patches ↓ prune B ↓ run A ↓ tests fail ↓ critic revises A ↓ tests pass ↓ final verifier passes ↓ production bug The visible failure is the production bug.
But where was the earliest divergence?
Maybe patch B was correct.
If so:
branch B correct ↓ scorer ranks B below A ↓ B pruned The incident is primarily a selection failure, not a generation failure.
That distinction matters enormously.
If you respond by making generation more expensive, you may increase cost while leaving the actual bug untouched.
This is why Step 12 introduced metrics such as oracle@N and selection regret.
Incident forensics brings those ideas down to a single production run.
Ask:
Was a valid solution present? If yes:
generation succeeded selection failed If no:
generation failed That one test can completely change the remediation.
6. A Practical Failure Taxonomy Use failure labels that point toward engineering actions.
class FailureStage(str, Enum): INTERPRETATION = "interpretation" EVIDENCE = "evidence" STATE = "state" ROUTING = "routing" PLANNING = "planning" GENERATION = "generation" SELECTION = "selection" SEARCH = "search" TOOL = "tool" EXECUTION = "execution" MEMORY = "memory" CRITIQUE = "critique" ESCALATION = "escalation" SCHEDULING = "scheduling" DISTRIBUTED_OWNERSHIP = "distributed_ownership" VERIFICATION = "verification" RELEASE = "release" CONTAINMENT = "containment" Then add a more specific reason.
STATE / STALE_REPOSITORY_SNAPSHOT ROUTING / WRONG_SPECIALIST SELECTION / CORRECT_BRANCH_PRUNED TOOL / SILENT_PARTIAL_RESULT CRITIQUE / CORRECT_TO_WRONG VERIFICATION / WRONG_ARTIFACT_BOUND DISTRIBUTED_OWNERSHIP / STALE_WORKER_COMMIT SCHEDULING / VERIFICATION_STARVATION RELEASE / INCOMPATIBLE_MEMORY_SCHEMA CONTAINMENT / RETRY_AMPLIFICATION Avoid labels such as:
LLM error agent error hallucination bad reasoning They are too broad to drive remediation.
7. Model Failure vs Evidence Failure One of the most common false diagnoses is blaming the model for missing information it never received.
Suppose the agent answers from retrieved documentation.
The answer is wrong.
Investigate in this order:
Was the authoritative evidence available? ↓ Was it retrieved? ↓ Was it selected into context? ↓ Did the model interpret it correctly? ↓ Did the verifier check against it? These are different failures.
Evidence unavailable The source did not contain the needed information.
Retrieval failure The source contained it, but retrieval did not surface it.
Context selection failure Retrieval found it, but the context builder dropped it.
Model interpretation failure The correct evidence was in context, but the model still chose incorrectly.
Verification failure The correct evidence existed, but the verifier did not use it.
Only one of those is primarily a model-generation problem.
8. State Failure vs Tool Failure Suppose an agent calls:
git diff and receives a correct result for an outdated worktree.
The tool worked.
The state was wrong.
Similarly:
browser fetch succeeded does not mean:
browser observed the intended page state You need to track:
tool correctness state identity freshness scope postcondition A useful forensic record:
@dataclass(frozen=True) class ObservationEvidence: observation_id: str source: str state_id: str observed_at: str freshness_ms: int scope: str payload_hash: str authoritative: bool Then you can distinguish:
TOOL_FAILURE from:
STATE_STALENESS That distinction feeds directly into Step 17’s uncertainty decomposition.
9. Routing Failure vs Expert Failure Suppose a mixture-of-agents system routes a database issue to a general coding model.
The result fails.
There are at least two possible incidents:
router should have selected DB expert or:
DB expert would also have failed Those require different fixes.
Use counterfactual replay where possible.
original route → FAIL DB expert shadow replay → PASS That supports a routing-failure hypothesis.
But be precise.
A historical counterfactual is not automatically ground truth if the alternative expert is evaluated under different state or evidence.
Use the same frozen input bundle whenever possible.
@dataclass(frozen=True) class CounterfactualResult: original_decision_id: str alternative: str replay_mode: str same_evidence: bool same_state: bool verified_outcome: str caveat: str | None = None 10. Candidate Failure vs Selection Failure This deserves its own forensic branch.
For any run that selected among alternatives, ask:
Did any rejected candidate pass the external verifier? If yes:
selection failure If no:
candidate-generation failure You can quantify the incident with selection regret.
best available verified utility - selected verified utility For a binary verifier:
rejected candidate = PASS selected candidate = FAIL selection regret = maximal That is far more actionable than saying:
MCTS failed Maybe MCTS generated the right branch and your scorer killed it.
11. Critic Failure Is a State Transition Do not evaluate a critic only by whether its output sounds better.
Track transitions.
wrong → correct correct → correct wrong → wrong correct → wrong The dangerous transition is:
correct → wrong During forensics, compare the pre-critic and post-critic candidate with the same external verifier.
@dataclass(frozen=True) class CriticTransition: before_candidate: str after_candidate: str before_verified: str after_verified: str critic_version: str If an incident includes:
PASS candidate ↓ critic revision ↓ FAIL candidate then the critic is not a neutral helper.
It is an incident participant.
12. Verifier Failure Can Make Every Other Component Look Healthy This is one of the most dangerous incident classes.
Suppose:
candidate wrong verifier PASS release promoted production failure later Every dashboard based on verifier PASS rate may claim the system was healthy.
So forensic analysis must independently ask:
Was the verifier evidence sufficient? Was it bound to the exact candidate? Was the verifier checking the right contract? Was the verifier version itself recently changed? Would an independent verifier disagree? Bind verifier evidence to exact candidate identity.
candidate_hash = abc123 verifier_evidence.candidate_hash = abc123 If those differ, you have a provenance failure before you even discuss semantics.
Then independently audit the verifier.
A verifier outage is visible.
A verifier that confidently approves the wrong thing is much worse.
13. Distributed Failures Need Ownership Forensics Step 20 introduced leases, heartbeats, idempotency, and fencing.
Incident forensics should reconstruct the ownership timeline.
worker A acquires epoch 41 worker A stalls lease expires worker B acquires epoch 42 worker B commits worker A resumes worker A attempts commit with epoch 41 Correct behavior:
worker A rejected Incident behavior:
worker A mutation accepted That is not a model failure.
It is a fencing enforcement failure.
Preserve:
operation_id attempt_id worker_id lease_epoch lease_acquired_at lease_expired_at commit_epoch mutation_result The side-effect boundary is where ownership authority must be proven.
14. Scheduler Failures Can Masquerade as Intelligence Failures Suppose the agent stops early with a weak answer.
You might conclude:
model could not solve task But the scheduler may have:
exhausted search nodes on duplicate branches, escalated too early, spent the verification reserve on speculative work, cancelled a promising slow branch, hit a tenant quota, entered degraded mode due to unrelated platform pressure. Reconstruct the budget ledger.
initial token budget initial tool budget initial search-node budget protected verifier reserve actual spend by component cancellations unused budget scheduler decisions Then ask:
Was the failure caused by insufficient capability, or by poor allocation of available capability?
That is a completely different remediation path.
15. Find the Earliest Incorrect Decision A practical algorithm:
1. Start from the externally observed failure. 2. Walk backward through causal dependencies. 3. At each decision, ask whether its inputs were correct. 4. If inputs were wrong, continue upstream. 5. If inputs were correct but the decision was wrong, mark divergence. 6. Test whether an alternative decision under the same evidence would pass. 7. Stop when you find the earliest supported divergence. Pseudo-code:
def find_earliest_divergence(events, verifier): ordered = causal_topological_order(events) for event in ordered: if not event.is_decision: continue if not inputs_are_valid(event): continue if decision_is_consistent_with_contract(event): continue alternatives = event.recorded_alternatives for alt in alternatives: if verifier(alt) == "PASS": return { "decision_id": event.id, "failure": "decision_failure", "better_alternative": alt.id, } return { "decision_id": event.id, "failure": "decision_failure_no_known_good_alternative", } return None In real systems, many checks will return UNKNOWN.
That is fine.
A forensic system that preserves uncertainty is better than one that invents causality.
16. Root Cause Needs Evidence Strength Not all incident conclusions deserve the same confidence.
Use explicit evidence levels.
class EvidenceStrength(str, Enum): DIRECT = "direct" STRONG = "strong" MODERATE = "moderate" WEAK = "weak" UNKNOWN = "unknown" Examples:
Direct Recorded candidate B passed the same external verifier. Recorded scorer pruned B. Strong Counterfactual replay under the same frozen evidence passes with route X. Moderate Similar incidents disappear when retrieval freshness improves. Weak A reviewer believes the model probably misunderstood the task. Do not promote weak stories into strong root-cause claims because they sound plausible.
17. Build an Incident Hypothesis Table For complex incidents, track competing explanations.
Hypothesis Supporting Evidence Contradicting Evidence Test Status Wrong route DB expert replay passes Generalist sometimes succeeds Frozen-input route replay Supported Stale retrieval Source changed recently Retrieved chunk hash current Compare snapshot timestamp Rejected Weak verifier Independent verifier fails Original verifier passes Gold-case audit Supported Model incapable Generalist fails repeatedly Same model passes with correct evidence Same-model evidence replay Rejected This prevents the investigation from locking onto the first plausible narrative.
18. Blast Radius Is Part of Root Cause Analysis A single failed run might indicate:
one corrupt task or:
all tasks using verifier v17 Those are very different incidents.
Use the provenance graph to query affected cohorts.
release_id model_version prompt_hash router_version verifier_version retrieval_snapshot policy_version tool_version tenant workload class Then ask:
How many historical runs share the suspect component? How many were independently verified? How many show the same failure signature? How many consequential side effects occurred? A blast-radius record might look like:
@dataclass(frozen=True) class BlastRadius: incident_id: str suspect_version: str affected_runs: int affected_tenants: int side_effecting_runs: int independently_failed_runs: int first_seen_at: str last_seen_at: str The investigation is not complete until you know whether you found a local anomaly or a systemic defect.
19. Group Incidents by Failure Signature Do not investigate every run independently if the same causal signature repeats.
A signature might include:
failure_stage=selection scorer_version=7 route=code candidate_margin<0.05 correct_candidate_pruned=true or:
failure_stage=verification verifier_version=12 artifact_binding=mismatch Cluster by structured evidence, not by vague semantic similarity alone.
This can reveal:
one bad release affecting thousands of runs, one tenant-specific schema mismatch, one model/tool combination with high failure rate, one browser domain whose semantics changed, one scheduler mode that starves verifiers. 20. Counterfactual Replay: Would the Fix Have Prevented the Incident? A remediation is not proven because it sounds reasonable.
Suppose the proposed fix is:
increase beam width from 4 to 8 Replay the incident.
If:
beam 8 still prunes the correct branch then the remediation did not address the root cause.
Maybe the scorer is the problem.
A strong remediation test looks like:
original trajectory → FAIL same frozen evidence + candidate policy fix → PASS Then run the fix against non-incident baselines.
incident cases improve normal cases do not regress materially cost remains acceptable false-success rate does not increase That is much stronger evidence.
21. Counterfactuals Have Limits Be careful.
You usually do not know the true outcome of an action that was never executed.
If the browser agent did not submit alternative form sequence B, you cannot claim with certainty that B would have succeeded against the historical external system.
So counterfactual replay should report:
VERIFIED SIMULATED UNOBSERVED UNKNOWN Do not manufacture historical facts.
For deterministic offline tasks, counterfactual evidence can be strong.
For mutable external systems, it may be much weaker.
22. Remediation Should Target the Earliest Controllable Failure The earliest divergence is not always controllable.
Example:
external API returned incorrect data You may not control the API.
But perhaps you control:
lack of cross-check or:
failure to preserve UNKNOWN or:
high-risk action allowed with single-source evidence A useful remediation targets the earliest controllable point that prevents or contains the incident.
uncontrollable trigger ↓ first controllable defense ↓ containment This is often more practical than trying to eliminate every upstream failure.
23. Fix Prevention, Detection, and Containment Separately A mature remediation often includes three classes of change.
Prevention Stop the bad trajectory from occurring.
Example:
reject stale repository snapshots before patch generation Detection Identify it earlier.
Example:
verify candidate artifact hash before PASS Containment Limit damage if prevention fails.
Example:
require human approval for production mutation when state freshness is UNKNOWN One fix rarely covers all three.
24. An Agent Incident Report Should Be Reproducible A useful report is not a story written from memory.
It is a structured claim backed by evidence.
Incident ID Run ID Behavioral release Task contract Observed impact External verifier result Timeline Causal graph Earliest divergence Failure stage Root cause Trigger Amplifiers Containment failures Detection failures Blast radius Evidence strength Counterfactual tests Remediation Regression tests Rollback / release action Residual uncertainty Every strong claim should point to a replay artifact, event, observation, or verifier result.
25. Example: Coding Agent Incident Suppose the task is:
Fix duplicate invoice generation without modifying authentication code. The agent returns PASS.
Later, authentication tests fail in production.
Forensics finds:
1. Repository snapshot was correct. 2. Planner proposed payments-only scope. 3. Search generated candidates A, B, C. 4. B modified payments only and passed all relevant tests. 5. A modified payments + auth. 6. Scorer ranked A slightly above B. 7. B was pruned. 8. A passed payment tests. 9. Scope verifier used planned scope rather than actual changed files. 10. Final verifier returned PASS. This incident contains at least two important failures.
Root cause SELECTION / CORRECT_BRANCH_PRUNED Detection failure VERIFICATION / ACTUAL_DIFF_NOT_CHECKED A remediation might include:
- include actual changed-file set in verifier input - require scope PASS before final PASS - tune scorer using selection-regret incidents Increasing model size would not address either root cause.
26. Example: Research Agent Incident Task:
Summarize the latest regulatory position on a technical standard. The answer cites an outdated draft.
Forensics:
1. Authoritative final standard existed. 2. Retrieval index snapshot was six weeks old. 3. Model accurately summarized retrieved draft. 4. Critic agreed with model. 5. Verifier checked citation presence, not source freshness. Root cause:
EVIDENCE / STALE_RETRIEVAL_SNAPSHOT Detection failure:
VERIFICATION / NO_FRESHNESS_REQUIREMENT The model was not the primary failure.
27. Example: Browser Agent Incident Task:
Cancel subscription renewal. The agent times out after clicking cancel, retries, and reports failure.
Later the account shows two cancellation-related requests and one billing inconsistency.
Forensics:
1. First click returned timeout. 2. External side effect status was ambiguous. 3. Retry policy treated timeout as failure. 4. Second attempt repeated the mutation. 5. No idempotency key existed. 6. No authoritative state check ran before retry. Root cause:
EXECUTION / AMBIGUOUS_MUTATION_RETRIED Containment failure:
NO_IDEMPOTENCY_BOUNDARY Correct remediation:
UNKNOWN after ambiguous timeout ↓ observe authoritative account state ↓ retry only if cancellation absent Not:
ask a stronger model what to do 28. Example: DevOps Agent Incident Task:
Restart one unhealthy worker group. During platform pressure, the agent restarts a healthy group instead.
Forensics:
1. Health data from dependency A was stale. 2. Scheduler entered SEVERE mode. 3. Freshness diagnostic was skipped to save latency. 4. Agent acted on stale state. 5. Verification happened after restart. Root cause:
SCHEDULING / HIGH_VALUE_DIAGNOSTIC_SKIPPED Contributing cause:
STATE / STALE_HEALTH_SNAPSHOT The remediation belongs partly in Step 18’s information policy and partly in Step 21’s overload policy.
29. Incident Metrics Worth Tracking Aggregate incident counts are not enough.
Track:
mean time to detect mean time to localize earliest divergence mean time to contain mean time to verify remediation replay completeness rate unknown-cause rate false root-cause rate repeat-incident rate blast radius per incident selection-failure rate verifier-failure rate state-staleness incident rate retry-amplification incident rate rollback success rate One particularly useful metric is:
repeat incident after remediation If that remains high, your forensics process may be producing plausible stories rather than real causal fixes.
30. Incident Forensics Should Feed Policy Calibration Incidents are high-value labelled trajectories.
But do not blindly train on them.
Feed structured evidence into the systems from Steps 14–18.
Examples:
selection incidents → scorer calibration routing incidents → router policy calibration stale-state incidents → state uncertainty policy verification incidents → verifier coverage policy budget incidents → scheduler calibration Keep the evidence lineage.
A postmortem conclusion should not magically become a training label with no provenance.
31. Do Not Let the Agent Write Its Own Root Cause Unchecked An LLM can summarize an incident graph.
It can propose hypotheses.
It can cluster similar incidents.
It can suggest likely causal chains.
But the root-cause record should remain evidence-backed.
A useful pattern:
LLM hypothesis ↓ structured evidence query ↓ replay / counterfactual test ↓ external verifier ↓ human or policy approval for high-impact conclusion Do not replace causal evidence with persuasive prose.
32. Build a Forensic Query Layer You want questions such as:
Show all runs using verifier v12 where PASS candidate hash did not match the verified artifact hash. Show all failed coding runs where a rejected candidate later passed offline verification. Show all incidents where stale state exceeded five minutes and the agent performed a side effect. Show all retries after ambiguous browser mutation timeouts. Show all runs cancelled under scheduler v8 where the cancelled branch later passed replay. This turns provenance into an operational debugging system rather than archival storage.
33. A Minimal Incident-Forensics Pipeline You do not need a giant platform to start.
A useful first version can be:
1. Preserve replay manifest. 2. Reconstruct causal event graph. 3. Identify external failure contract. 4. Walk backward to earliest divergence. 5. Classify failure stage. 6. Check rejected alternatives. 7. Check verifier binding and coverage. 8. Measure blast radius by version/cohort. 9. Run bounded counterfactual tests. 10. Record prevention/detection/containment fixes. 11. Add regression case. 12. Verify remediation before promotion. That already provides far more value than:
read transcript ask model why it failed change prompt 34. Incident Regression Cases Become a Reliability Asset Every confirmed incident should create a regression artifact.
@dataclass(frozen=True) class IncidentRegressionCase: incident_id: str frozen_input_bundle: str expected_contract: str original_release_id: str original_outcome: str remediation_release_id: str expected_remediated_outcome: str verifier_version: str Then future releases must answer:
Does this release still prevent the incident? This turns production failures into durable evidence.
35. But Do Not Overfit to the Incident Set Incident cases are important, but they are not the whole workload.
A fix can prevent one incident while damaging ordinary cases.
So every remediation should be evaluated against:
incident regression set normal held-out set high-risk cohort set cost/latency baseline false-success baseline UNKNOWN baseline This connects directly to Step 12’s compute-matched benchmarking and Step 24’s promotion gates.
36. Incident Severity Should Reflect Consequence, Not Drama A verbose model failure with no side effects may be low severity.
A quiet verifier false PASS before a production mutation may be critical.
Severity should consider:
external impact side-effect irreversibility number of affected runs number of affected tenants security/safety implication verification bypass financial impact recoverability Do not derive severity from how surprising the transcript looks.
37. Preserve UNKNOWN Root Cause When Evidence Is Missing Sometimes you cannot determine the cause.
The model provider no longer exposes the same version.
The retrieval snapshot was not preserved.
The browser response body was discarded.
The external system changed.
The verifier evidence was not bound to the candidate.
Then the correct root-cause state may be:
UNKNOWN with a secondary finding:
FORENSIC_GAP / REQUIRED_ARTIFACT_NOT_RETAINED That itself is an actionable reliability defect.
Do not fill evidence gaps with certainty.
38. Make Forensic Completeness a Release Requirement If a component can influence consequential agent behavior, its decisions should be reconstructable.
That means release gates should ask:
Can we identify which version made this decision? Can we reconstruct its inputs? Can we retrieve its output artifact? Can we bind downstream verification to that artifact? Can we determine whether it produced side effects? Can we replay it safely? If not, the component may be operationally opaque even if it performs well in benchmarks.
39. A Strong Incident Forensics Invariant Set Useful invariants include:
Every consequential side effect has an operation ID. Every distributed mutation records ownership epoch. Every final PASS references verifier evidence. Every verifier evidence record references the exact candidate hash. Every routing decision records alternatives and policy version. Every pruning decision records branch identity and reason. Every external observation records state identity/freshness when applicable. Every release ID resolves to an immutable manifest. Every replay declares whether it uses recorded or live observations. Every counterfactual declares whether the alternative outcome was observed or simulated. Every root-cause claim has an evidence-strength classification. These are boring.
That is why they are useful.
40. The Full Reliability Loop At this point the Advanced Agents series has moved far beyond “how do I make an LLM call tools?”
The production loop now looks like this:
task ↓ explicit behavioral release ↓ platform admission ↓ per-run scheduling ↓ typed uncertainty ↓ value-of-information decisions ↓ search / tools / specialists / verification ↓ distributed ownership + fencing ↓ side effects ↓ external verification ↓ trajectory + provenance ↓ behavioral drift monitoring ↓ incident detection ↓ forensic replay ↓ earliest divergence ↓ root-cause evidence ↓ counterfactual remediation test ↓ incident regression case ↓ release promotion gate That is what an advanced agent architecture begins to look like when you treat it as software rather than a demo.
Do You Actually Need an Incident-Forensics Pipeline? Not always.
If your application is:
single-user read-only low consequence fully observable cheap to rerun then ordinary structured logs may be enough.
But incident forensics becomes valuable when you have:
multiple models or experts, routing, search, tool use, mutable external state, retries, distributed workers, side effects, dynamic policies, release rollouts, verifiers, regulated or auditable workflows, costly failures. The more mechanisms that can influence the final result, the less useful “look at the transcript” becomes.
Final Principle The most important rule is simple:
Do not stop at the component that produced the visible failure. Find the earliest evidence-backed divergence, identify the first controllable defense that should have prevented or contained it, and prove the remediation against the original incident trajectory.
That gives you a much stronger engineering loop than prompt tweaking.
FAIL ↓ reconstruct ↓ localize ↓ classify ↓ test competing hypotheses ↓ measure blast radius ↓ replay remediation ↓ add regression case ↓ promote only with evidence And that sets up the next problem.
Once incident evidence exists across thousands of runs, how do you turn those incidents into a systematic reliability program rather than a pile of postmortems?
That is the next stage: agent SLOs, error budgets, reliability targets, and deciding which classes of failure deserve engineering effort first.