# Advanced Agents From First Principles 15: How Do You Optimize an Agent Policy Without Turning It Into Another Black Box?
# Book: Advanced Agents From First Principles
# Chapter URL: https://programmer.ie/books/advanced-agents-from-first-principles/15-chapter/
# Prose fingerprint (sha256, normalized): ed5140168feb9409f9595a4c3c265e6aba240a981c92544d9f5e265328a57226
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

How Do You Optimize an Agent Policy Without Turning It Into Another Black Box? By now our advanced agent can do a lot.
It can:
route tasks to different models or specialists, decide whether to search, choose a search budget, decide when to escalate, invoke critics, retry or recover, stop when evidence is strong enough, learn from verified production trajectories, and trace the decisions that produced each outcome. That creates a new problem.
The runtime itself now contains a policy.
Not just the language model.
The orchestration layer is deciding things such as:
Should I use the cheap model or the expensive one? Should I search or answer directly? How many branches should I expand? Should I invoke a critic? Should I retry? Should I escalate? Should I stop? Those decisions can dominate both quality and cost.
So the obvious next step is to optimize them.
And this is where advanced agent systems can quietly become another black box.
A team starts with a few explicit thresholds.
Then they add a learned router.
Then a learned search controller.
Then another LLM decides whether the first LLM should call the second LLM.
Eventually nobody can explain why a task used 27 model calls instead of 3.
The control system becomes harder to reason about than the model it was supposed to manage.
The central rule of this post is:
Optimize the agent policy, but keep the control layer simpler than the system it controls whenever possible.
We want policies that are:
measurable, calibratable, interpretable, versioned, reversible, and easy to compare against a fixed baseline. That usually means starting much smaller than another general-purpose LLM.
1. The Agent Runtime Already Has a Policy Consider a production coding agent.
A request arrives:
Fix the flaky test in payments/tests/test_retry.py The runtime may make several decisions before any code is changed:
request ↓ route task ↓ cheap model or strong model? ↓ plan directly or search? ↓ run one candidate or several? ↓ invoke critic? ↓ execute tests ↓ retry / escalate / stop? Each arrow is a policy decision.
We can represent the runtime abstractly as:
state s ↓ policy π(s) ↓ action a ↓ new state s' ↓ external evidence The language model may generate the content of an action.
But the runtime policy decides which computation to buy.
That distinction matters.
A bad answer may not mean the model was incapable.
It may mean:
the router chose the wrong model, search was skipped when it was needed, search continued too long, escalation happened too late, the critic was invoked unnecessarily, the runtime stopped before verification, or the system spent its budget on the wrong branch. These are orchestration-policy failures.
2. What Exactly Are We Optimizing? A common mistake is to optimize one scalar such as:
reward = success - cost That can produce ugly behavior.
If the penalty on cost is too strong, the system may stop early and increase false-success rates.
If success dominates everything, the runtime may spend enormous compute on every task.
If latency is ignored, the system may become unusable even while benchmark success improves.
A better framing is constrained optimization.
For example:
minimize expected_cost subject to: verified_success >= 0.95 false_success <= 0.01 p95_latency <= 30 seconds safety_violations == 0 This changes the question.
Instead of:
What policy maximizes a mysterious reward?
we ask:
What is the cheapest policy that satisfies the reliability requirements?
That is much easier to reason about.
3. Start With Explicit Policies Before training anything, write down the current control policy.
For example:
from dataclasses import dataclass @dataclass class TaskFeatures: estimated_complexity: float verifier_available: bool prior_failure_rate: float tool_count: int context_tokens: int def choose_model(x: TaskFeatures) -> str: if not x.verifier_available: return "strong" if x.estimated_complexity > 0.75: return "strong" if x.prior_failure_rate > 0.20: return "strong" return "cheap" This policy is not sophisticated.
That is a strength.
We can inspect it.
We can log why it routed a task.
We can replay old trajectories through it.
We can compare candidate thresholds.
We can roll back instantly.
And most importantly, it creates a baseline.
If a learned router cannot beat this simple policy under equal conditions, we do not need the learned router.
4. Policy Optimization Begins With Features A policy can only be as good as the signals it sees.
Useful runtime features might include:
task type repo size files touched estimated change scope number of tools available verifier availability historical failure rate retrieval confidence critic disagreement search score margin previous retries budget consumed elapsed time current UNKNOWN rate But feature design creates a dangerous temptation.
Teams often feed the entire conversation into another LLM and ask:
Should we escalate? That may work.
It is also expensive, hard to calibrate, difficult to replay deterministically, and often unnecessary.
A smaller policy may be enough:
features ↓ logistic regression ↓ P(failure if cheap model) ↓ threshold ↓ cheap / strong Or:
features ↓ decision tree ↓ search width = 1 / 3 / 5 Or simply:
(task_type, verifier_available) -> policy table The rule is simple:
Use the smallest policy class that captures the decision boundary you actually need.
5. Optimize One Decision at a Time Do not train one giant controller to decide everything.
Separate the decisions.
For example:
router policy search policy critic policy escalation policy stopping policy Each has different evidence.
Router policy Question:
Which expert/model should handle this task? Evidence:
expert-specific success rates, latency, cost, task features, verifier outcomes. Search policy Question:
How much search should this task receive? Evidence:
oracle@N, pruning regret, marginal success gain per extra node, branch diversity, cost. Critic policy Question:
Is critique likely to improve this candidate? Evidence:
wrong -> correct correct -> correct wrong -> wrong correct -> wrong Escalation policy Question:
Should we pay for a stronger model? Evidence:
rescue rate, unnecessary escalation rate, expected extra cost, verifier uncertainty. Stopping policy Question:
Do we have enough evidence to stop? Evidence:
verifier state, confidence interval, remaining unresolved criteria, marginal gain from another step. These policies should not automatically share the same model or objective.
6. Threshold Tuning Is Often Enough Suppose the runtime produces a probability that the cheap model will fail:
P(failure | task) = 0.31 The policy is:
if failure_probability > threshold: escalate() else: continue_cheap() The difficult part is not the classifier.
It is choosing the threshold.
We can evaluate candidate thresholds over held-out trajectories:
threshold success cost/task escalation rate ---------------------------------------------------- 0.20 97.1% $0.091 42% 0.30 96.8% $0.074 31% 0.40 95.9% $0.061 22% 0.50 93.8% $0.050 14% If our reliability requirement is 96%, then 0.30 may dominate 0.20.
It achieves the target with lower cost.
No reinforcement-learning system is required.
No controller LLM is required.
Just calibrated predictions plus an explicit threshold.
7. Calibration Matters More Than Raw Accuracy Suppose a router predicts:
0.9 probability specialist A succeeds If tasks assigned 0.9 actually succeed only 65% of the time, the policy is badly calibrated.
That matters because runtime policies often make threshold decisions.
A classifier with slightly lower raw accuracy but good calibration may be more useful.
Useful diagnostics include:
reliability diagrams Brier score expected calibration error precision / recall by route cost-weighted routing regret For a search controller:
predicted value of another branch vs actual marginal success improvement For escalation:
predicted rescue probability vs observed rescue probability A policy should know how uncertain it is.
8. Add an Explicit Abstain Region A policy does not always need to decide.
Suppose the router estimates:
P(cheap succeeds) = 0.51 P(strong succeeds) = 0.54 The difference is tiny.
Instead of pretending the classifier knows, we can define:
if margin < 0.05: use conservative fallback Or:
if uncertainty > threshold: route to verifier-backed path This creates a third state:
cheap strong ABSTAIN / FALLBACK That is often safer than forcing every prediction into a binary decision.
It mirrors the UNKNOWN principle from verification.
9. Search Budget Is a Policy Decision Search should not always use the same width and depth.
A runtime could choose among:
no search Best-of-3 beam width 3 beam width 5 MCTS 20 nodes MCTS 50 nodes The naive approach is:
hard task -> more search But difficulty alone is not enough.
Search only helps when additional computation can discriminate between alternatives.
Useful features include:
candidate score margin candidate disagreement verifier availability historical oracle@N branch diversity remaining budget search-depth improvement curve A simple policy might be:
def choose_search_budget(score_margin: float, oracle_gain: float) -> int: if score_margin > 0.25: return 1 if oracle_gain < 0.03: return 1 if score_margin > 0.10: return 3 return 5 Again, this is intentionally boring.
Boring control logic is good when it works.
10. Learn Marginal Value, Not Just Success For search and escalation, the question is not:
Will this task succeed? The better question is:
Will spending more compute improve the outcome enough to justify the cost? For another search expansion:
marginal_value = P(success after expansion) - P(success now) Then compare that against cost.
Conceptually:
if expected_gain > required_gain_for_cost: continue else: stop This is much better than a fixed max_steps = 20 for every task.
11. Model Selection Should Be Cost-Aware Imagine three models:
local-small $0.001/task cloud-medium $0.02/task frontier-large $0.20/task Their verified success rates may differ by task class.
For simple extraction:
local-small 98% cloud-medium 99% frontier-large 99% The frontier model adds almost nothing.
For difficult code repair:
local-small 45% cloud-medium 72% frontier-large 91% Now escalation may be justified.
A routing policy should optimize expected cost subject to required reliability, not blindly maximize capability.
12. Routing Regret A useful metric is routing regret.
For a task \(i\):
regret_i = cost/chance penalty of chosen route relative to best feasible route Suppose:
cheap model succeeds for $0.01 strong model also succeeds for $0.20 Routing to strong incurred unnecessary cost.
If:
cheap fails strong succeeds routing to cheap may incur reliability regret.
A cost-aware regret function can encode both.
The important point is that router quality is not just classification accuracy.
A cheap mistake and an expensive mistake are not equivalent.
13. Offline Policy Evaluation We do not want to deploy every candidate policy just to see what happens.
The observability work from Step 13 gives us recorded trajectories.
The trajectory-learning work from Step 14 gives us verified outcomes and policy lineage.
Now we can replay decisions offline.
For each historical decision state:
state features available actions actual chosen action observed outcome counterfactual evidence where available A candidate policy can be evaluated against that dataset.
But there is a warning.
Historical data is biased by the old policy.
If the old router almost never selected expert B, we may have weak evidence about B.
So offline evaluation must distinguish:
observed evidence counterfactual replay simulated evidence unknown evidence Do not silently turn missing counterfactuals into assumed outcomes.
14. Pairwise Policy Comparison A practical pattern is to compare policies directly.
policy A: current production policy B: candidate Replay both across the same task set.
Measure:
verified success cost/task p50 latency p95 latency false-success rate UNKNOWN rate route distribution search calls escalation rate Then compute per-task deltas.
This is much more informative than comparing two aggregate averages from different runs.
15. Policy Optimization Is Not Model Fine-Tuning These ideas are easy to confuse.
Model fine-tuning Changes the model’s parameters.
prompt -> model weights -> output Runtime policy optimization Changes how the system allocates computation.
state -> control policy -> model/tool/search decision You can optimize the runtime policy without touching the foundation model.
That is often cheaper, safer, and easier to reverse.
16. Keep Safety Outside the Learned Policy A learned policy must not decide fundamental authorization boundaries.
Do not train a controller to decide whether it may:
bypass authentication, access another tenant, disable mandatory verification, execute production mutations without approval, ignore credential scope, escape a sandbox. Those are invariants.
The adaptive policy may choose among allowed actions.
It should not redefine what is allowed.
safety envelope ↓ allowed action set ↓ optimized policy ↓ selected action Not:
optimized policy ↓ maybe obey safety 17. Version the Policy Like Code Every production decision should identify the policy version that made it.
For example:
{ "policy_family": "model_router", "policy_version": "router-2026-08-09-03", "feature_schema": "router_features_v5", "thresholds": { "escalate": 0.31, "abstain_margin": 0.05 } } Then a trajectory can say:
this route was selected by router-2026-08-09-03 That gives us:
reproducibility, rollback, differential analysis, canary comparison, and auditability. 18. Treat Policy Configuration as an Artifact Do not hide important runtime behavior in scattered constants.
A policy artifact might look like:
model_router: version: router-v7 cheap_model: local-qwen strong_model: frontier failure_threshold: 0.31 abstain_margin: 0.05 search_policy: version: search-v4 default_width: 1 uncertain_width: 3 maximum_width: 5 min_expected_gain: 0.04 critic_policy: version: critic-v2 invoke_when_score_below: 0.72 stopping_policy: version: stop-v5 require_verifier_pass: true max_unknown_retries: 1 Now architecture changes are explicit and reviewable.
19. Policy Diffing Suppose a new production version suddenly costs 35% more.
Without policy lineage, debugging may take hours.
With policy artifacts:
-search.maximum_width: 3 +search.maximum_width: 5 -model_router.failure_threshold: 0.38 +model_router.failure_threshold: 0.27 The likely cause is immediately visible.
Policy configuration should be diffable for the same reason code is diffable.
20. Do Not Optimize Everything Simultaneously If you simultaneously change:
model router, search budget, critic invocation, escalation threshold, stopping policy, and performance improves, you do not know why.
Worse, one change may be helping while another is hurting.
Prefer controlled experiments.
baseline ↓ change router only ↓ measure ↓ change search policy only ↓ measure This is slower than architectural enthusiasm.
It is much faster than debugging a giant adaptive system later.
21. Feature Ablation A learned policy can become more complicated than necessary.
Suppose the router uses:
17 features Remove them one at a time.
If removing prompt_length changes nothing, delete it.
If removing verifier_available causes a major regression, keep it.
The same evidence-first principle applies to features.
feature exists ↓ measurable contribution? ├── yes -> keep └── no -> delete 22. Policy Complexity Is a Cost A more complex controller adds:
latency, training burden, monitoring burden, calibration drift, more feature dependencies, harder replay, harder rollback, harder explanation. So policy complexity belongs in the cost function.
A tiny decision tree that achieves 96.5% verified success may be preferable to a neural controller achieving 96.7%.
The difference may not earn the operational burden.
23. Example: Coding Agent Model Router Imagine a coding agent has two models.
small model strong model Features:
files likely affected presence of tests historical task-class failure rate stack depth repo size whether task touches migrations whether task touches authentication A simple policy could be:
def route_code_task( files_estimate: int, tests_present: bool, touches_migrations: bool, touches_auth: bool, ) -> str: if touches_auth or touches_migrations: return "strong" if files_estimate > 5: return "strong" if not tests_present: return "strong" return "small" After enough verified trajectories, maybe a logistic model improves cost without reducing success.
Great.
But the explicit rule remains the baseline.
24. Example: Research Agent Search Budget A research system may need more search when sources disagree.
Features:
number of independent sources source disagreement claim importance source freshness primary-source availability Policy:
low disagreement -> stop moderate disagreement -> retrieve 3 more sources high disagreement + high-impact claim -> escalate research depth The goal is not maximal retrieval.
The goal is enough evidence to reach the required confidence.
25. Example: DevOps Escalation Policy A DevOps agent may begin in read-only diagnostic mode.
Possible stages:
local diagnostics ↓ read-only cluster inspection ↓ stronger model analysis ↓ human approval ↓ mutation The adaptive policy may optimize when to request more diagnostics or a stronger model.
It should not optimize away the human approval boundary for dangerous mutations.
The safety envelope remains fixed.
26. Example: Browser Agent Retry Policy A browser agent fails to submit a form.
Possible causes:
DOM changed field validation failed session expired wrong selector network timeout A naive policy retries the same action.
A better policy uses the failure class.
network timeout -> bounded retry selector missing -> re-observe DOM validation error -> inspect field state session expired -> re-auth flow This is policy optimization through failure-specific control, not generic persistence.
27. Example: Mixture-of-Agents Runtime A mixture-of-agents system may choose among:
direct answer specialist model Best-of-N critic/revision Tree of Thoughts MCTS multi-agent debate frontier escalation Do not ask one monolithic controller to choose among everything from day one.
Use staged routing.
For example:
task ↓ cheap difficulty classifier ├─ easy -> direct ├─ medium -> specialist / Best-of-N └─ hard -> advanced path ↓ search-needed classifier ├─ no -> strong model └─ yes -> search policy Hierarchical control keeps each decision narrow.
28. Policy Cascades A useful production pattern is a cascade.
cheap path ↓ fail/uncertain specialist path ↓ fail/uncertain bounded search ↓ fail/uncertain strong model ↓ fail/uncertain UNKNOWN / human review Each stage should have:
an entry condition, a cost, an expected rescue probability, an exit condition, and a verifier. The policy decides whether expected rescue justifies the next stage.
29. Measure Rescue Rate For an escalation stage:
rescue_rate = failures_before_stage_that_become_PASS / stage_invocations But rescue rate alone is not enough.
Also measure:
cost per rescue latency per rescue false-success change UNKNOWN reduction A stage that rescues 5% of tasks but adds 10x latency to all tasks may not be worth it.
30. Counterfactual Policy Questions Trajectory observability lets us ask questions such as:
What if we had not escalated? What if beam width had been 3 instead of 5? What if the critic had been skipped? What if this task had used specialist B? What if we had stopped after verifier stage 1? These are much more useful than:
Did the final run pass? They isolate the value of a policy decision.
31. Policy Promotion Pipeline A candidate policy should pass several stages.
candidate ↓ offline replay ↓ held-out benchmark ↓ shadow mode ↓ small canary ↓ production At every stage compare against the current production policy.
A policy is not promoted because it looks clever.
It is promoted because it improves the required metrics.
32. Shadow Mode In shadow mode the candidate policy makes decisions but does not control production behavior.
For each real request:
production policy -> actual action candidate policy -> shadow action Now we can inspect disagreement.
route disagreement rate search-budget disagreement escalation disagreement predicted cost delta predicted success delta High disagreement deserves investigation before canary deployment.
33. Canary Policy Deployment After shadow validation:
95% production policy 5% candidate policy Measure:
verified success, false-success rate, UNKNOWN rate, cost, p95 latency, policy-specific failure modes. Promotion should require explicit gates.
For example:
verified_success >= baseline - 0.2% cost <= baseline - 8% false_success <= baseline p95_latency <= baseline + 5% The exact numbers depend on the application.
The important point is that the promotion rule is explicit.
34. Rollback Must Be Boring If a new policy causes regressions, rollback should be trivial.
active_policy = router-v8 ↓ regression active_policy = router-v7 Do not require retraining.
Do not require reconstructing old prompts.
Do not require reverse-engineering a controller LLM.
A reversible system learns faster because experiments are less dangerous.
35. Policy Drift Even a good policy may become wrong over time.
Reasons include:
model upgrades, tool changes, task-distribution shifts, new repositories, verifier changes, price changes, latency changes, new experts. Monitor:
route distribution expert utilization calibration error cost per verified success policy disagreement success by task class A threshold calibrated six months ago may no longer be appropriate.
36. Model Upgrades Can Break the Router Suppose the cheap model improves dramatically.
The old router may still send 40% of tasks to the expensive model.
Nothing is technically broken.
But the policy is now wasteful.
This is why model version belongs in policy evaluation.
policy effectiveness = f(policy, model versions, task distribution) Policies cannot be evaluated independently from the capabilities they route between.
37. Price Changes Can Change the Optimal Policy Suppose the strong model becomes 80% cheaper.
The reliability frontier may not change.
The economically optimal routing threshold can.
Agent architecture is partly an economics problem.
The optimal policy is not static when resource prices change.
38. Deterministic Policy Beats Learned Policy More Often Than People Expect Some tasks have clear rules.
For example:
if production mutation -> require human approval if no verifier available -> strong model if migration file touched -> run migration verifier if tests fail -> do not report PASS These should remain deterministic.
Do not replace known invariants with statistical prediction.
Use learning where the boundary is genuinely uncertain.
39. Keep the Learned Surface Small A strong architecture often looks like:
deterministic safety rules ↓ deterministic workflow structure ↓ small learned routing/search policies ↓ LLMs used for generative reasoning ↓ external verification Not:
LLM decides everything The runtime should absorb what can be made explicit.
The model should handle the parts that actually benefit from generative reasoning.
40. The Policy Should Be Explainable in One Record When the runtime makes a decision, log something like:
{ "decision": "escalate_model", "policy_version": "escalation-v4", "features": { "cheap_failure_probability": 0.43, "verifier_unknown": true, "retries": 1 }, "threshold": 0.31, "chosen_action": "frontier_model", "alternatives": ["retry_cheap", "stop_unknown"], "expected_extra_cost": 0.17 } You do not need hidden reasoning.
You need operational evidence.
41. Build a Small Policy Runtime Here is a compact provider-agnostic structure.
from dataclasses import dataclass from typing import Any, Mapping @dataclass(frozen=True) class PolicyDecision: policy_name: str policy_version: str action: str reason: str features: Mapping[str, Any] score: float | None = None class Policy: name: str version: str def decide(self, features: Mapping[str, Any]) -> PolicyDecision: raise NotImplementedError class ThresholdEscalationPolicy(Policy): name = "escalation" def __init__(self, version: str, threshold: float): self.version = version self.threshold = threshold def decide(self, features: Mapping[str, Any]) -> PolicyDecision: p_fail = float(features["cheap_failure_probability"]) if p_fail > self.threshold: action = "strong_model" reason = f"{p_fail:.3f} > {self.threshold:.3f}" else: action = "cheap_model" reason = f"{p_fail:.3f} <= {self.threshold:.3f}" return PolicyDecision( policy_name=self.name, policy_version=self.version, action=action, reason=reason, features=dict(features), score=p_fail, ) The point is not the code.
The point is that policy decisions become explicit objects.
They can be:
logged, replayed, tested, benchmarked, diffed, and rolled back. 42. Test Policies Like Ordinary Software A threshold policy should have unit tests.
def test_escalates_above_threshold(): policy = ThresholdEscalationPolicy("v1", threshold=0.3) d = policy.decide({"cheap_failure_probability": 0.7}) assert d.action == "strong_model" def test_stays_cheap_below_threshold(): policy = ThresholdEscalationPolicy("v1", threshold=0.3) d = policy.decide({"cheap_failure_probability": 0.1}) assert d.action == "cheap_model" Then add dataset-level tests:
verified success >= target cost <= budget false-success <= limit A learned policy does not exempt the runtime from ordinary software engineering.
43. Test Policy Invariants Useful invariants include:
never mutate production without authorization never claim PASS without required verifier evidence never exceed hard cost budget never route outside allowed model set never use stale policy artifact never mix tenant feature data These are stronger than statistical performance metrics.
They must hold on every run.
44. Monitor Policy Disagreement If a candidate and production policy agree 99.8% of the time, the candidate probably cannot change much.
If they disagree 45% of the time, deployment risk is high.
Disagreement rate is therefore useful during policy development.
But disagreement alone is not good or bad.
We need to know whether disagreement occurs on the right tasks.
45. Measure Decision-Level Value For each policy action, track outcomes.
Example:
Decision: invoke critic invocations: 10,000 wrong -> correct: 1,420 correct -> wrong: 310 unchanged wrong: 2,100 unchanged correct: 6,170 Net correction:
1,420 - 310 = +1,110 Now include cost.
If critic calls are extremely expensive, the policy may still need tighter targeting.
46. Do Not Confuse Correlation With Policy Value Suppose difficult tasks are more likely to use MCTS.
Those tasks also have lower success rates.
A naive analysis could conclude:
MCTS reduces success That may be false.
The system routes MCTS to harder problems.
This is selection bias.
Use:
matched task groups, counterfactual replay, randomized exploration where safe, shadow policies, controlled benchmark strata. Trajectory data is powerful, but causal attribution still matters.
47. Controlled Exploration If a policy never tries an alternative, it cannot learn whether the alternative improved.
For low-risk decisions we can allocate small exploration traffic.
For example:
95% current best route 5% safe alternative But exploration must respect safety and cost limits.
Do not randomly explore dangerous production mutations.
Use exploration for decisions such as:
model routing, search width, critic invocation, retrieval depth, safe diagnostic sequences. 48. Exploration Has a Budget Too Track:
exploration cost exploration success delta new information gained regressions caused Exploration is not free just because it helps learning.
49. When Should You Use a Learned Controller? A learned policy becomes more attractive when:
the decision boundary is genuinely fuzzy, many verified trajectories exist, features are stable, the action set is narrow, counterfactual evaluation is possible, mistakes are observable, rollback is easy. It is less attractive when:
data is sparse, safety depends on the decision, the environment changes constantly, outcomes are weakly verified, the action space is huge, the learned policy is harder to understand than the problem. 50. A Useful Escalation Ladder for Policy Complexity Start here:
hard-coded rule If insufficient:
threshold table Then:
small decision tree Then:
logistic regression / calibrated classifier Then perhaps:
gradient-boosted model / small neural policy Only then ask whether an LLM controller is justified.
This is the same architecture principle we have used throughout the series:
Complexity must earn its place.
51. The Agent Control Plane At this point we can see the runtime as two layers.
Execution plane models tools memory search critics verifiers Control plane routing policy search policy escalation policy critic policy stopping policy budget policy The control plane decides which execution-plane capabilities are used.
This is a useful architectural boundary.
It allows execution mechanisms to improve without rewriting the orchestration logic from scratch.
52. Policy Optimization Can Simplify the Execution Plane Suppose the telemetry shows:
critic invoked 30% of runs net verified improvement: 0.2% latency increase: 18% Delete the critic.
Suppose MCTS is used on 12% of tasks but only beats beam search on one narrow category.
Route only that category to MCTS.
Suppose the frontier model is called on 45% of tasks but the cheap model now performs equally well on half of them.
Raise the escalation threshold.
Optimization should often produce:
less architecture not more.
53. The Complete Evidence Loop We now have a closed system.
production tasks ↓ agent runtime ↓ decision traces ↓ external verification ↓ trajectory dataset ↓ attribution ↓ candidate policy ↓ offline replay ↓ compute-matched benchmark ↓ shadow ↓ canary ↓ promotion / rollback ↓ production tasks That loop is much more important than any individual agent technique.
It gives us a way to improve the runtime without guessing.
54. Ten Rules for Agent Policy Optimization If you remember only ten things, remember these.
Optimize decisions, not architecture aesthetics. Use external verification as the outcome signal. Optimize cost and latency subject to reliability constraints. Start with explicit rules and thresholds. Optimize routing, search, critics, escalation and stopping separately. Calibrate probabilities before trusting thresholds. Keep safety invariants outside the learned policy. Version every policy and feature schema. Promote through replay, shadow and canary stages. Delete mechanisms that fail to earn their cost. 55. Where We Are Now The series began with advanced orchestration techniques.
We then built toward something more important.
A disciplined agent runtime.
We now have:
advanced mechanisms ↓ architecture selection ↓ compute-matched benchmarking ↓ trajectory observability ↓ verified trajectory learning ↓ policy optimization The result is not an agent that endlessly adds more intelligence-shaped machinery.
It is a system that can answer:
What failed? Why did it fail? Which mechanism was supposed to fix it? Did that mechanism actually help? How much did it cost? Can we route it more selectively? Can we simplify the architecture? That is the difference between an impressive demo and an engineering system.
The next stage is to go one level deeper into the control plane itself: how to design budget allocation as a first-class resource scheduler across models, search branches, tools and verifiers rather than relying on fixed limits everywhere.