# Advanced Agents From First Principles 28: Where Should You Spend the Next Engineering Hour? Prioritize Reliability by Risk and Expected Return
# Book: Advanced Agents From First Principles
# Chapter URL: https://programmer.ie/books/advanced-agents-from-first-principles/28-chapter/
# Prose fingerprint (sha256, normalized): 7dad50715da6ccf1c1d835e47c1ad297fb99491cdb1d5fc5f81faef525ba94ea
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

Advanced agent systems can fail in a hundred different ways.
The model can hallucinate.
The router can choose the wrong expert.
Retrieval can surface stale evidence.
A verifier can become too permissive.
A critic can turn a correct answer into a wrong one.
A scheduler can spend all its budget on search and starve verification.
A browser worker can replay an irreversible action.
A distributed worker can lose its lease and still try to commit.
A release can be technically healthy while semantic quality quietly drifts.
Once you have observability, replay, incident forensics, SLOs and error budgets, the next problem is not finding possible work.
It is deciding which work matters most.
That sounds obvious until you have ten plausible reliability projects and only enough engineering capacity for two.
Should you:
improve the model, add a stronger verifier, tune the router, improve retrieval, add more tests, tighten tool schemas, redesign memory, add another critic, increase search depth, reduce search depth, add a fallback model, isolate a browser pool, improve incident detection, reduce agent authority, or simplify the architecture entirely? The wrong answer is:
Work on whichever failure feels most intellectually interesting.
The second-worst answer is:
Work on whichever component has the most incidents.
The useful question is:
Which engineering change is expected to reduce the most important verified reliability loss per unit of engineering effort and operational cost?
That is the core rule for this post.
The Search Problem: “How Do I Prioritize AI Agent Reliability Work?” Most production agent teams eventually accumulate a backlog that looks something like this:
- router misclassifies some coding tasks - verifier misses malformed tool outputs - retrieval index is occasionally stale - expensive model escalation happens too often - browser runs time out under load - critic sometimes degrades correct answers - search duplicates equivalent branches - UNKNOWN rate increased after stricter verification - one high-risk workflow had a false PASS Every item is real.
Every item can consume engineering time.
But they are not equally important.
A frequent low-severity failure can matter less than a rare false success in a high-authority workflow.
A technically impressive fix can matter less than a one-line permission reduction.
A model upgrade can improve generation while increasing tool misuse, latency and verification cost.
A better verifier may reduce reported PASS rate while making the system more reliable.
Raw incident count is therefore a weak prioritization signal.
So is average success rate.
So is user complaint volume.
So is benchmark score in isolation.
You need a prioritization model that begins with reliability consequence.
Start From the Reliability Loss, Not the Component Step 27 introduced agent SLOs and error budgets.
That means the system already measures outcomes such as:
verified_success false_success UNKNOWN latency cost_per_verified_success verifier_coverage side_effect_integrity Those are better starting points than architecture labels.
Suppose you observe this monthly error-budget consumption:
false_success budget burn 62% ordinary failure budget burn 31% UNKNOWN budget burn 18% latency budget burn 44% cost budget burn 73% side-effect integrity burn 5% The first question is not:
Which component is most sophisticated?
It is:
Which reliability dimensions are consuming the most important budget?
That immediately changes the engineering conversation.
If cost is burning quickly but false-success risk is healthy, the right fix may be search pruning or model routing.
If false success is burning quickly, reducing compute cost is secondary.
If verifier coverage is weak, improving generation may not even be measurable yet.
The reliability vector tells you where the system is hurting.
Incident forensics tells you why.
Prioritization combines the two.
Do Not Collapse Reliability Into One Magic Score It is tempting to create something like:
reliability_score = success * 0.5 - false_success * 5.0 - cost * 0.1 - latency * 0.05 That may be useful for a narrow optimization experiment.
It is dangerous as the only management signal.
Why?
Because weighted scores hide constraints.
Imagine:
change A: +8% verified success +0.5% false success change B: +3% verified success -0.2% false success If false success is safety-critical, change A may be unacceptable no matter how good its weighted score looks.
A stronger formulation is constrained:
maximize expected verified_success gain subject to: false_success <= ceiling side_effect_integrity >= floor verifier_coverage >= floor p95_latency <= limit average_cost <= budget Now engineering optimization cannot buy apparent quality by violating the reliability contract.
Build a Remediation Candidate, Not Just a Ticket A normal backlog ticket might say:
Improve router accuracy That is too vague for reliability prioritization.
A useful remediation candidate should record:
from dataclasses import dataclass from typing import Optional @dataclass(frozen=True) class RemediationCandidate: id: str failure_class: str target_slo: str affected_cohort: str expected_incident_reduction: float expected_false_success_reduction: float expected_cost_reduction: float expected_latency_reduction_ms: float confidence: float engineering_days: float rollout_risk: float reversibility: float evidence_ref: str proposed_change: str The point is not that these estimates are perfectly precise.
They will not be.
The point is to force the team to write down:
what failure is being targeted, what reliability budget it affects, which cohort is affected, what improvement is expected, how strong the evidence is, what the implementation cost is, how risky the rollout is, and whether the change is reversible. That is already far better than ranking work by intuition.
A Practical Expected Reliability Return A useful first approximation is:
expected_reliability_return = expected_loss_reduction × confidence × affected_volume × severity_weight Then divide by implementation and operational cost:
priority ≈ expected_reliability_return --------------------------------- engineering_effort + rollout_cost This is deliberately simple.
The important thing is that the numerator is tied to verified reliability loss.
Not architectural novelty.
Not model benchmark prestige.
Not incident count alone.
Severity Matters More Than Frequency Alone Suppose two incident classes exist.
Failure A retrieval misses useful context frequency: 400 / month outcome: task returns UNKNOWN Failure B browser agent confirms purchase twice frequency: 2 / month outcome: duplicate external side effect Raw frequency says A is 200 times more common.
Reliability consequence may say B is far more urgent.
This is why the prioritization model needs severity tiers.
For example:
S0 = catastrophic / irreversible / safety-critical S1 = high-impact false success or external side effect S2 = task failure requiring user recovery S3 = degraded quality or extra latency S4 = internal inefficiency only The exact scale is less important than consistency.
A two-incident S1 class can outrank a thousand S4 inefficiencies.
False Success Deserves a Disproportionate Weight A normal failure is visible.
The system says:
FAIL or:
UNKNOWN A false success is different.
The system says:
PASS while reality says:
wrong That is especially dangerous because downstream systems may trust the output.
So reliability prioritization should usually give false-success reduction a stronger weight than ordinary failure reduction.
For example:
ordinary task failure prevented = 1 reliability unit UNKNOWN reduced safely = 0.5 reliability units false PASS prevented = 10 reliability units unsafe side effect prevented = 25 reliability units Do not copy those numbers blindly.
The important concept is asymmetric consequence.
Blast Radius Changes Priority Suppose a router bug affects:
2% of code-review tasks and a verifier bug affects:
all workflows using verifier v17 Even if both create the same per-run error probability, their system-wide impact differs.
Blast radius can be estimated across dimensions such as:
release version model version prompt version router version verifier version retrieval version tool version tenant risk class workflow class A simple impact model is:
expected_loss = probability_of_failure × affected_volume × consequence_per_failure This is much more useful than saying:
We had six incidents involving retrieval.
Recurrence Probability Matters Some incidents are one-off combinations of unusual state.
Others reveal a stable failure mode.
Forensics should estimate recurrence separately from severity.
You might record:
incident: wrong repository selected severity: high recurrence evidence: low incident: verifier ignores missing test evidence severity: high recurrence evidence: high Both are serious.
The second likely deserves earlier systematic remediation because it remains latent across many future runs.
Detection Quality Changes the Economics A failure that is immediately detected is different from a failure that remains silent.
Compare:
bad candidate ↓ verifier rejects ↓ FAIL with:
bad candidate ↓ verifier approves ↓ PASS ↓ external damage Even if the original model error is identical, the second path is more dangerous because containment failed.
So remediation priorities should include detection coverage.
A useful risk factor is:
undetected_risk = failure_probability × probability_failure_escapes_detection × consequence This often makes verifier work surprisingly valuable.
The Highest-Value Fix May Not Be the Root Cause Fix Imagine an incident chain:
retrieval misses authoritative source ↓ model generates wrong answer ↓ critic approves ↓ verifier misses unsupported claim ↓ false PASS The root cause may be retrieval.
But the highest-value engineering change may be verifier hardening.
Why?
Because the verifier may catch:
retrieval failures, model failures, critic failures, stale-memory failures, and future unknown failure modes. This is a critical distinction:
Root-cause priority and remediation priority are not always the same thing.
The best remediation is the one that reduces the most important future loss at acceptable cost.
Prevention, Detection and Containment Have Different Returns Step 26 separated remediation into:
prevention detection containment That distinction becomes very useful here.
For the same incident, you might have three candidate projects.
Prevention Improve the router so the wrong expert is not selected.
Detection Add an independent verifier that notices the expert’s output is invalid.
Containment Prevent the workflow from committing external side effects unless verification passes.
The containment fix may be cheapest and most powerful.
Or detection may protect against a broad class of future failures.
Or prevention may remove expensive downstream recovery entirely.
The prioritization system should compare these instead of assuming root-cause prevention automatically wins.
Engineering Effort Is Not Just Coding Time A candidate change may look cheap to implement but expensive to deploy safely.
Engineering cost should include:
implementation benchmark construction migration work shadow evaluation canary monitoring operational complexity on-call burden rollback complexity ongoing maintenance For example:
add another critic LLM may take one afternoon to code.
But it can add:
model cost, latency, more failure modes, more trace volume, more calibration work, more rollout surface, more correlated reasoning. The real cost is not one afternoon.
This is why the series keeps returning to the same principle:
Advanced complexity must earn its operational cost.
Prefer Changes With Broad Reliability Coverage Some fixes target one narrow symptom.
Others create a reliability layer that catches many classes of error.
Compare:
prompt patch for one hallucination pattern with:
source-grounding verifier for all research answers The prompt patch may be faster.
The verifier may have much broader coverage.
You can represent this as:
coverage = number_of_failure_classes_materially_reduced But do not maximize coverage blindly either.
A broad mechanism that is weak everywhere may be worse than a narrow deterministic guard for a high-severity failure.
Deterministic Fixes Often Have Better Reliability Economics Suppose an agent sometimes writes files outside an allowed directory.
Candidate A:
prompt: "Be very careful to only modify allowed files." Candidate B:
if not path.is_relative_to(allowed_root): raise PermissionError(path) Candidate B is usually superior because it provides:
lower uncertainty lower inference cost lower latency stronger guarantee easier testing clearer failure semantics Reliability prioritization should therefore include mechanism strength.
A deterministic invariant can dominate a probabilistic behavioral fix even when both target the same incident.
The Best Reliability Project Can Be Reducing Agent Authority This deserves special emphasis.
Teams often assume reliability work means making the agent smarter.
Sometimes the cheapest high-return fix is reducing what the agent is allowed to do autonomously.
Suppose a deployment agent has authority to:
build approve and deploy If false-success risk is high, you could spend months improving the model.
Or you could change the authority boundary:
agent may build agent may propose deployment independent verifier must approve human or deterministic gate commits The model did not become smarter.
The system became safer.
That can be the highest-return reliability change available.
Authority Is a Reliability Variable We can model authority as part of consequence.
For example:
read-only research agent false result → bad recommendation browser purchasing agent false result → external transaction production deployment agent false result → system outage The same model error probability has radically different expected loss.
So remediation priority should consider:
authority_level reversibility side_effect_scope human_recovery_cost Reducing authority can reduce expected loss immediately while deeper quality improvements continue.
Use Incident Clusters, Not Individual Stories One dramatic incident can distort planning.
Instead, Step 26’s incident signatures should be clustered.
For example:
cluster A: stale repository state 42 incidents ordinary failure cluster B: permissive verifier after prompt change 3 incidents false PASS cluster C: browser timeout after external mutation 8 incidents ambiguous side-effect state Now remediation decisions can be based on repeated failure structure rather than anecdotes.
Build a Reliability Opportunity Table A practical planning table might look like this:
Candidate Target Expected loss reduction Confidence Effort Rollout risk Priority strengthen verifier binding false PASS high high low low very high retrain router routing failures medium medium high medium medium add third critic candidate quality low low medium medium low reduce browser authority side effects high high low low very high increase MCTS nodes generation failures uncertain low high low low The point is not the exact formula.
The point is that every project must state its expected reliability return.
Confidence Matters You may believe a fix will reduce failures by 50%.
But what supports that belief?
Useful evidence levels might be:
DIRECT counterfactual replay proves the fix prevented the incident STRONG repeated incidents share the same mechanism and offline replay improves them MODERATE correlated evidence with plausible causal support WEAK architecture intuition only A large theoretical benefit with weak evidence should often rank below a smaller well-supported improvement.
That is especially true for expensive architecture changes.
Use Counterfactual Remediation Replay Step 26 introduced the idea that a remediation should be replayed against the original incident.
That becomes a core prioritization input.
For each candidate fix:
original incident ↓ replay with remediation ↓ would failure still occur? Then run it against a broader incident set:
incident corpus ↓ remediation candidate ↓ prevented incidents regressed incidents unchanged incidents Now the expected benefit is grounded in evidence.
Do Not Overfit to the Incident Corpus A remediation that fixes every known incident can still harm normal traffic.
So evaluate:
incident regression suite + held-out normal workload + new task cohort + cost baseline + latency baseline + false-success baseline + UNKNOWN baseline You need to know both:
how much historical failure is prevented? and:
what new failure is introduced? Reliability Return Should Be Cohort-Aware Suppose a change improves coding-agent reliability by 10% overall.
That sounds good.
But perhaps:
small repositories: +14% large monorepos: -6% high-risk migrations: -9% The aggregate improvement hides a dangerous regression.
So expected benefit should be computed by cohort:
expected_return = Σ( cohort_volume × cohort_risk_weight × expected_improvement ) This connects directly to Step 27’s cohort-specific SLOs.
Marginal Return Matters More Than Historical Return Imagine retrieval quality has improved repeatedly:
project 1: +18% reliability project 2: +7% project 3: +2% The fourth retrieval project may have diminishing return.
Meanwhile the verifier may still have an obvious high-value gap.
Reliability planning should compare the next unit of investment, not celebrate historical importance.
The decision is:
Where does the next engineering hour have the highest expected reliability return?
Not:
Which subsystem has historically mattered most?
Opportunity Cost Is Real Engineering capacity is finite.
Choosing project A means delaying B.
So ranking should include opportunity cost.
Suppose:
Project A expected false-PASS reduction: 40% effort: 2 days Project B expected ordinary-failure reduction: 15% effort: 6 weeks Even if B is architecturally larger, A may be the obviously superior near-term reliability investment.
Use Portfolios, Not Just One Ranking There is one complication.
The highest-ranked project may not be enough by itself.
Reliability work can have dependencies.
For example:
better verifier telemetry ↓ enables verifier drift detection ↓ enables safe verifier optimization So planning can use a small portfolio:
1 immediate containment fix 1 high-confidence reliability improvement 1 enabling observability project This avoids spending all capacity on either short-term patches or long-term platform work.
A Simple Reliability Portfolio Model You can classify candidates into four groups.
1. Immediate containment Reduce high-severity risk now.
Examples:
reduce authority add hard guard block unsafe workflow require approval 2. High-confidence remediation Fix proven recurrent causes.
Examples:
correct stale-state check repair verifier binding fix router rule 3. Measurement improvement Increase ability to detect and attribute failures.
Examples:
add verifier coverage logging add branch lineage improve incident signatures 4. Capability investment Improve the underlying agent where evidence says it is the limiting factor.
Examples:
stronger model better retrieval better search specialist agent The order matters.
A capability investment is often premature if containment and measurement are still weak.
Model Quality Is Only One Investment Category Suppose generation failures account for 12% of reliability loss.
But verifier failures account for 45% of false-success loss.
Buying a stronger model may be the wrong move.
This is exactly why the architecture should not be model-centric.
The system’s reliability may be constrained by:
state quality retrieval routing tool semantics selection verification scheduling distributed coordination release discipline authority boundaries Model improvement should compete with every other remediation on evidence.
Search More Only If Search Failure Is the Bottleneck Advanced agents often respond to failure by increasing search.
More candidates.
Wider beam.
More MCTS nodes.
More critics.
More debate.
But if the real reliability loss is:
selection failure verifier failure stale state wrong route more search can make things worse by increasing cost and the number of wrong candidates presented to a weak selector.
Search investment only earns priority when incident forensics shows candidate-generation scarcity is actually limiting verified success.
Add a “Delete It” Candidate Every reliability planning exercise should include one candidate that asks:
What if we remove the mechanism?
Examples:
remove critic remove planner remove speculative branch remove second model remove memory promotion remove autonomous mutation Then benchmark:
current architecture vs simplified architecture If reliability stays flat while cost and failure surface fall, simplification wins.
This is one of the most important recurring ideas in this series:
The optimal advanced-agent architecture can become less advanced over time.
Estimate Reliability Return With Ranges, Not Fake Precision Do not write:
expected benefit = 17.43% unless you genuinely have enough data to support that precision.
Prefer ranges:
expected false-success reduction: 20–35% confidence: moderate or distributions if you have enough evidence.
The goal is decision support, not numerical theatre.
Track Predicted Versus Actual Return Once a remediation ships, compare prediction to reality.
Record:
@dataclass(frozen=True) class RemediationOutcome: candidate_id: str predicted_loss_reduction_low: float predicted_loss_reduction_high: float actual_loss_reduction: float predicted_effort_days: float actual_effort_days: float new_regressions: int rollback_required: bool Over time this improves prioritization itself.
You may learn that:
router fixes are consistently overestimated verifier fixes have strong cross-cutting value model upgrades cost more operationally than expected authority reductions are unusually effective Now the prioritization process becomes empirically calibrated.
Reliability Prioritization Can Drift Too Your prioritization policy is itself a policy.
It can become biased.
For example:
team overweights visible incidents team underweights UNKNOWN team ignores low-frequency false PASS effort estimates are systematically optimistic large architecture projects receive prestige bias So version the prioritization policy.
A simple record might include:
policy_version severity_weights false_success_multiplier cohort_weights confidence_scale effort_model rollout_risk_penalty Then compare whether the planning policy actually predicts realized reliability improvement.
Do Not Let the Agent Prioritize Its Own Reliability Targets The agent may help summarize evidence.
It may cluster incidents.
It may estimate remediation options.
But the agent should not have authority to redefine:
false-success ceiling risk classes required verification acceptable side effects release freeze criteria Those are external reliability contracts.
Otherwise the optimizer can improve its score by changing the rules.
Coding-Agent Example Imagine a coding agent has these monthly incident clusters:
selection failure: 42 stale repository state: 31 false verifier PASS: 4 over-expensive model routing: 95 The naive order is:
routing selection state verifier because routing has the most incidents.
But suppose impact analysis says:
routing incidents: cost only selection incidents: visible FAIL state incidents: visible FAIL verifier incidents: incorrect changes merged Now the verifier may be first priority despite having only four incidents.
Possible portfolio:
1. harden verifier candidate/hash binding 2. add authoritative git-state refresh before execution 3. tune model routing for cost That ordering follows reliability consequence, not count.
Research-Agent Example Suppose a research agent has:
stale retrieval: high frequency unsupported claims: medium frequency slow source discovery: high frequency If unsupported claims are producing false PASS, the highest-value project may be:
citation/evidence verifier rather than a larger retrieval index.
The verifier can contain multiple upstream errors while retrieval improvements continue later.
Browser-Agent Example Suppose browser automation suffers from:
page-navigation retries DOM selector drift ambiguous purchase confirmation A model upgrade may improve selectors.
But the highest-value reliability change may be:
make purchase submission non-autonomous + verify authoritative transaction state after timeout That directly reduces side-effect risk.
DevOps-Agent Example Suppose a deployment agent has:
occasional bad remediation plan rare stale cluster state slow verifier The highest priority may be:
mandatory fresh cluster-state observation or:
fenced approval gateway before production mutation These can dominate a model-quality project because they reduce consequence even when reasoning still fails occasionally.
Build a Reliability Investment Review A useful weekly or release-cycle review can ask:
1. Which SLO budgets are burning fastest? 2. Which failure clusters consume those budgets? 3. What are the strongest evidence-backed causes? 4. What candidate remediations exist? 5. What reliability loss would each remove? 6. What is the evidence confidence? 7. What is the implementation + rollout cost? 8. What new risks could each change introduce? 9. Can authority reduction contain the risk sooner? 10. Which candidate has the highest marginal return? This is much more disciplined than:
What should we improve in the agent this sprint?
A Minimal Prioritizer You do not need a learned optimizer to start.
from dataclasses import dataclass @dataclass(frozen=True) class Candidate: name: str expected_loss_reduction: float confidence: float severity_multiplier: float affected_volume: float effort_days: float rollout_risk: float def priority(c: Candidate) -> float: benefit = ( c.expected_loss_reduction * c.confidence * c.severity_multiplier * c.affected_volume ) cost = max(c.effort_days * (1.0 + c.rollout_risk), 0.1) return benefit / cost Then keep hard constraints outside the score:
if candidate.increases_false_success: reject() if candidate.violates_safety_contract: reject() This is intentionally boring.
That is a feature.
When Should You Use a Learned Prioritizer? Probably later than you think.
Start with explicit rules and measured outcomes.
A learned prioritizer only becomes attractive when you have enough historical data connecting:
incident cluster remediation type predicted benefit predicted effort actual benefit actual effort regressions Even then, keep the output advisory first.
The planning system should not automatically approve high-risk architectural changes because a model predicts high ROI.
Reliability Economics Is Not Just Money The word “economics” here includes all scarce resources:
engineering time model compute latency budget verification capacity operational complexity on-call attention user trust external side-effect risk The most expensive reliability failure may consume almost no cloud spend.
A false success can destroy trust much faster than an inefficient model route.
The Goal Is Not Maximum Reliability at Any Cost A system can become unusable if reliability is pursued without regard to latency, cost or capability.
For example:
run 15 models run 6 critics search 500 nodes require 12 verifiers may reduce some error classes.
It may also make the product economically impossible.
The target is:
the required reliability for the workload at an acceptable cost and latency.
That is why SLOs are contracts rather than aspirations for perfection.
The Goal Is Also Not Maximum Capability The opposite failure is capability chasing.
Teams add:
more agents more models more search more memory more autonomy while reliability budgets burn.
Step 27 gives you a simple rule:
if reliability budget is healthy: experiment if reliability budget is burning: harden if critical budget is exhausted: freeze or reduce authority Step 28 adds:
when hardening: choose the highest expected reliability return per unit of real engineering cost The Architecture Is Now a Reliability Investment Portfolio At this stage in the series, the system contains many possible control surfaces:
model router retrieval search critic memory scheduler verifier distributed coordination release gates fallbacks circuit breakers authority boundaries None of them should be sacred.
Every mechanism must continue earning its place through evidence.
You may discover that:
model upgrade → small gain, high cost verifier hardening → large gain, low cost critic removal → no reliability loss, lower latency authority reduction → major risk reduction That is exactly the sort of result a mature agent platform should be able to discover.
A Useful Final Formula You can think of reliability investment as:
priority ≈ expected reduction in important verified loss × evidence confidence × recurrence × blast radius × severity ---------------------------------------------- implementation cost × rollout risk × operational burden Do not worship the formula.
Use it to force explicit assumptions.
Then validate those assumptions after the change ships.
What Should You Measure? For the prioritization system itself, measure:
predicted vs actual reliability gain predicted vs actual engineering effort predicted vs actual rollout risk fraction of projects that improve target SLO fraction introducing regressions rollback rate reliability gain per engineering day false-success reduction per engineering day cost reduction per engineering day latency reduction per engineering day And separately track whether the team keeps investing in low-return work despite the evidence.
That is a process failure worth observing too.
Failure Modes of Reliability Prioritization The prioritizer can fail in predictable ways.
1. Incident-count bias Frequent low-impact failures dominate attention.
2. Severity blindness Rare false successes are underweighted.
3. Root-cause fixation Teams assume the root-cause fix is always the best remediation.
4. Architecture prestige bias Large model/search projects outrank boring deterministic safeguards.
5. Effort optimism Rollout, migration and maintenance cost are omitted.
6. Measurement blindness Teams improve components they cannot reliably evaluate.
7. Aggregate-metric bias Overall success improves while a critical cohort regresses.
8. Authority blindness The team keeps improving autonomy instead of reducing dangerous authority.
9. Diminishing-return blindness The same subsystem receives investment long after its marginal return has fallen.
10. Self-scoring The optimized agent influences the reliability target used to judge itself.
Each of these should be visible in the planning process.
Deliberately Test the Prioritization Framework You can create synthetic planning exercises.
For example:
Scenario A: frequent cheap failure 1000 monthly failures low severity high detection cheap recovery Scenario B: rare false success 3 monthly failures high severity low detection external side effects Scenario C: expensive architecture proposal potentially large gain weak evidence high effort high rollout complexity Scenario D: boring deterministic guard moderate gain high evidence low effort very low rollout risk The prioritizer should not automatically choose the largest architecture project.
If it does, the policy is probably wrong.
A Mature Agent Team Should Be Able to Say No One of the signs of maturity is the ability to reject an attractive mechanism.
For example:
"MCTS is interesting, but our incident evidence says verifier coverage is the current reliability bottleneck." or:
"The frontier model improves generation, but the current router sends too many easy tasks to it. Fix routing first." or:
"The agent can probably be made safer eventually, but removing autonomous purchase authority eliminates the immediate high-severity risk today." That is engineering discipline.
The Reliability Loop The full loop now looks like this:
production runs ↓ verified outcomes ↓ SLO / error-budget consumption ↓ incident clusters ↓ causal forensics ↓ remediation candidates ↓ expected reliability return ↓ prioritized engineering work ↓ offline replay / benchmark ↓ shadow / canary / promotion ↓ actual reliability return ↓ update prioritization evidence That is a much stronger system than “improve the agent whenever users complain.”
Final Principle Advanced agent engineering eventually becomes a resource-allocation problem.
You have finite:
engineering time compute latency verification capacity operational attention risk budget You cannot maximize everything.
So make the trade-offs explicit.
Measure the reliability loss that actually matters.
Trace it to evidence-backed failure classes.
Generate competing remediation options.
Include prevention, detection, containment and authority reduction.
Estimate benefit with uncertainty.
Include real rollout and maintenance cost.
Prefer deterministic safeguards when they solve the problem cleanly.
Measure what happened after shipping.
And then ask the same question again:
Where should the next engineering hour go?
The answer should come from verified evidence—not architectural fashion.
Next Step 28 gives the platform a way to choose reliability investments.
The next stage is to make human escalation and authority boundaries explicit: when the agent should proceed autonomously, when it should ask for approval, when it should hand off with evidence, and how to design escalation so humans are not merely rubber-stamping opaque model decisions.