How Do You Optimize an Agent Policy Without Turning It Into Another Black Box?

By now our advanced agent can do a lot.

It can:

• route tasks to different models or specialists,
• decide whether to search,
• choose a search budget,
• decide when to escalate,
• invoke critics,
• retry or recover,
• stop when evidence is strong enough,
• learn from verified production trajectories,
• and trace the decisions that produced each outcome.

That creates a new problem.

The runtime itself now contains a policy.

Not just the language model.

The orchestration layer is deciding things such as:

Should I use the cheap model or the expensive one?
Should I search or answer directly?
How many branches should I expand?
Should I invoke a critic?
Should I retry?
Should I escalate?
Should I stop?

Those decisions can dominate both quality and cost.

So the obvious next step is to optimize them.

And this is where advanced agent systems can quietly become another black box.

A team starts with a few explicit thresholds.

Then they add a learned router.

Then a learned search controller.

Then another LLM decides whether the first LLM should call the second LLM.

Eventually nobody can explain why a task used 27 model calls instead of 3.

The control system becomes harder to reason about than the model it was supposed to manage.

The central rule of this post is:

Optimize the agent policy, but keep the control layer simpler than the system it controls whenever possible.

We want policies that are:

• measurable,
• calibratable,
• interpretable,
• versioned,
• reversible,
• and easy to compare against a fixed baseline.

That usually means starting much smaller than another general-purpose LLM.

1. The Agent Runtime Already Has a Policy

Consider a production coding agent.

A request arrives:

Fix the flaky test in payments/tests/test_retry.py

The runtime may make several decisions before any code is changed:

request
   ↓
route task
   ↓
cheap model or strong model?
   ↓
plan directly or search?
   ↓
run one candidate or several?
   ↓
invoke critic?
   ↓
execute tests
   ↓
retry / escalate / stop?

Each arrow is a policy decision.

We can represent the runtime abstractly as:

state s
   ↓
policy π(s)
   ↓
action a
   ↓
new state s'
   ↓
external evidence

The language model may generate the content of an action.

But the runtime policy decides which computation to buy.

That distinction matters.

A bad answer may not mean the model was incapable.

It may mean:

• the router chose the wrong model,
• search was skipped when it was needed,
• search continued too long,
• escalation happened too late,
• the critic was invoked unnecessarily,
• the runtime stopped before verification,
• or the system spent its budget on the wrong branch.

These are orchestration-policy failures.

2. What Exactly Are We Optimizing?

A common mistake is to optimize one scalar such as:

reward = success - cost

That can produce ugly behavior.

If the penalty on cost is too strong, the system may stop early and increase false-success rates.

If success dominates everything, the runtime may spend enormous compute on every task.

If latency is ignored, the system may become unusable even while benchmark success improves.

A better framing is constrained optimization.

For example:

minimize expected_cost
subject to:
    verified_success >= 0.95
    false_success <= 0.01
    p95_latency <= 30 seconds
    safety_violations == 0

This changes the question.

Instead of:

What policy maximizes a mysterious reward?

we ask:

What is the cheapest policy that satisfies the reliability requirements?

That is much easier to reason about.

3. Start With Explicit Policies

Before training anything, write down the current control policy.

For example:

from dataclasses import dataclass


@dataclass
class TaskFeatures:
    estimated_complexity: float
    verifier_available: bool
    prior_failure_rate: float
    tool_count: int
    context_tokens: int


def choose_model(x: TaskFeatures) -> str:
    if not x.verifier_available:
        return "strong"

    if x.estimated_complexity > 0.75:
        return "strong"

    if x.prior_failure_rate > 0.20:
        return "strong"

    return "cheap"

This policy is not sophisticated.

That is a strength.

We can inspect it.

We can log why it routed a task.

We can replay old trajectories through it.

We can compare candidate thresholds.

We can roll back instantly.

And most importantly, it creates a baseline.

If a learned router cannot beat this simple policy under equal conditions, we do not need the learned router.

4. Policy Optimization Begins With Features

A policy can only be as good as the signals it sees.

Useful runtime features might include:

task type
repo size
files touched
estimated change scope
number of tools available
verifier availability
historical failure rate
retrieval confidence
critic disagreement
search score margin
previous retries
budget consumed
elapsed time
current UNKNOWN rate

But feature design creates a dangerous temptation.

Teams often feed the entire conversation into another LLM and ask:

Should we escalate?

That may work.

It is also expensive, hard to calibrate, difficult to replay deterministically, and often unnecessary.

A smaller policy may be enough:

features
   ↓
logistic regression
   ↓
P(failure if cheap model)
   ↓
threshold
   ↓
cheap / strong

Or:

features
   ↓
decision tree
   ↓
search width = 1 / 3 / 5

Or simply:

(task_type, verifier_available) -> policy table

The rule is simple:

Use the smallest policy class that captures the decision boundary you actually need.

5. Optimize One Decision at a Time

Do not train one giant controller to decide everything.

Separate the decisions.

For example:

router policy
search policy
critic policy
escalation policy
stopping policy

Each has different evidence.

Router policy

Question:

Which expert/model should handle this task?

Evidence:

• expert-specific success rates,
• latency,
• cost,
• task features,
• verifier outcomes.

Search policy

Question:

How much search should this task receive?

Evidence:

• oracle@N,
• pruning regret,
• marginal success gain per extra node,
• branch diversity,
• cost.

Critic policy

Question:

Is critique likely to improve this candidate?

Evidence:

wrong -> correct
correct -> correct
wrong -> wrong
correct -> wrong

Escalation policy

Question:

Should we pay for a stronger model?

Evidence:

• rescue rate,
• unnecessary escalation rate,
• expected extra cost,
• verifier uncertainty.

Stopping policy

Question:

Do we have enough evidence to stop?

Evidence:

• verifier state,
• confidence interval,
• remaining unresolved criteria,
• marginal gain from another step.

These policies should not automatically share the same model or objective.

6. Threshold Tuning Is Often Enough

Suppose the runtime produces a probability that the cheap model will fail:

P(failure | task) = 0.31

The policy is:

if failure_probability > threshold:
    escalate()
else:
    continue_cheap()

The difficult part is not the classifier.

It is choosing the threshold.

We can evaluate candidate thresholds over held-out trajectories:

threshold    success    cost/task    escalation rate
----------------------------------------------------
0.20         97.1%      $0.091       42%
0.30         96.8%      $0.074       31%
0.40         95.9%      $0.061       22%
0.50         93.8%      $0.050       14%

If our reliability requirement is 96%, then 0.30 may dominate 0.20.

It achieves the target with lower cost.

No reinforcement-learning system is required.

No controller LLM is required.

Just calibrated predictions plus an explicit threshold.

7. Calibration Matters More Than Raw Accuracy

Suppose a router predicts:

0.9 probability specialist A succeeds

If tasks assigned 0.9 actually succeed only 65% of the time, the policy is badly calibrated.

That matters because runtime policies often make threshold decisions.

A classifier with slightly lower raw accuracy but good calibration may be more useful.

Useful diagnostics include:

reliability diagrams
Brier score
expected calibration error
precision / recall by route
cost-weighted routing regret

For a search controller:

predicted value of another branch
vs
actual marginal success improvement

For escalation:

predicted rescue probability
vs
observed rescue probability

A policy should know how uncertain it is.

8. Add an Explicit Abstain Region

A policy does not always need to decide.

Suppose the router estimates:

P(cheap succeeds) = 0.51
P(strong succeeds) = 0.54

The difference is tiny.

Instead of pretending the classifier knows, we can define:

if margin < 0.05:
    use conservative fallback

Or:

if uncertainty > threshold:
    route to verifier-backed path

This creates a third state:

cheap
strong
ABSTAIN / FALLBACK

That is often safer than forcing every prediction into a binary decision.

It mirrors the UNKNOWN principle from verification.

9. Search Budget Is a Policy Decision

Search should not always use the same width and depth.

A runtime could choose among:

no search
Best-of-3
beam width 3
beam width 5
MCTS 20 nodes
MCTS 50 nodes

The naive approach is:

hard task -> more search

But difficulty alone is not enough.

Search only helps when additional computation can discriminate between alternatives.

Useful features include:

candidate score margin
candidate disagreement
verifier availability
historical oracle@N
branch diversity
remaining budget
search-depth improvement curve

A simple policy might be:

def choose_search_budget(score_margin: float, oracle_gain: float) -> int:
    if score_margin > 0.25:
        return 1

    if oracle_gain < 0.03:
        return 1

    if score_margin > 0.10:
        return 3

    return 5

Again, this is intentionally boring.

Boring control logic is good when it works.

10. Learn Marginal Value, Not Just Success

For search and escalation, the question is not:

Will this task succeed?

The better question is:

Will spending more compute improve the outcome enough to justify the cost?

For another search expansion:

marginal_value =
    P(success after expansion)
    - P(success now)

Then compare that against cost.

Conceptually:

if expected_gain > required_gain_for_cost:
    continue
else:
    stop

This is much better than a fixed max_steps = 20 for every task.

11. Model Selection Should Be Cost-Aware

Imagine three models:

local-small     $0.001/task
cloud-medium    $0.02/task
frontier-large  $0.20/task

Their verified success rates may differ by task class.

For simple extraction:

local-small     98%
cloud-medium    99%
frontier-large  99%

The frontier model adds almost nothing.

For difficult code repair:

local-small     45%
cloud-medium    72%
frontier-large  91%

Now escalation may be justified.

A routing policy should optimize expected cost subject to required reliability, not blindly maximize capability.

12. Routing Regret

A useful metric is routing regret.

For a task \(i\):

regret_i = cost/chance penalty of chosen route
           relative to best feasible route

Suppose:

cheap model succeeds for $0.01
strong model also succeeds for $0.20

Routing to strong incurred unnecessary cost.

If:

cheap fails
strong succeeds

routing to cheap may incur reliability regret.

A cost-aware regret function can encode both.

The important point is that router quality is not just classification accuracy.

A cheap mistake and an expensive mistake are not equivalent.

13. Offline Policy Evaluation

We do not want to deploy every candidate policy just to see what happens.

The observability work from Step 13 gives us recorded trajectories.

The trajectory-learning work from Step 14 gives us verified outcomes and policy lineage.

Now we can replay decisions offline.

For each historical decision state:

state features
available actions
actual chosen action
observed outcome
counterfactual evidence where available

A candidate policy can be evaluated against that dataset.

But there is a warning.

Historical data is biased by the old policy.

If the old router almost never selected expert B, we may have weak evidence about B.

So offline evaluation must distinguish:

observed evidence
counterfactual replay
simulated evidence
unknown evidence

Do not silently turn missing counterfactuals into assumed outcomes.

14. Pairwise Policy Comparison

A practical pattern is to compare policies directly.

policy A: current production
policy B: candidate

Replay both across the same task set.

Measure:

verified success
cost/task
p50 latency
p95 latency
false-success rate
UNKNOWN rate
route distribution
search calls
escalation rate

Then compute per-task deltas.

This is much more informative than comparing two aggregate averages from different runs.

15. Policy Optimization Is Not Model Fine-Tuning

These ideas are easy to confuse.

Model fine-tuning

Changes the model’s parameters.

prompt -> model weights -> output

Runtime policy optimization

Changes how the system allocates computation.

state -> control policy -> model/tool/search decision

You can optimize the runtime policy without touching the foundation model.

That is often cheaper, safer, and easier to reverse.

16. Keep Safety Outside the Learned Policy

A learned policy must not decide fundamental authorization boundaries.

Do not train a controller to decide whether it may:

• bypass authentication,
• access another tenant,
• disable mandatory verification,
• execute production mutations without approval,
• ignore credential scope,
• escape a sandbox.

Those are invariants.

The adaptive policy may choose among allowed actions.

It should not redefine what is allowed.

safety envelope
      ↓
allowed action set
      ↓
optimized policy
      ↓
selected action

Not:

optimized policy
      ↓
maybe obey safety

17. Version the Policy Like Code

Every production decision should identify the policy version that made it.

For example:

{
  "policy_family": "model_router",
  "policy_version": "router-2026-08-09-03",
  "feature_schema": "router_features_v5",
  "thresholds": {
    "escalate": 0.31,
    "abstain_margin": 0.05
  }
}

Then a trajectory can say:

this route was selected by router-2026-08-09-03

That gives us:

• reproducibility,
• rollback,
• differential analysis,
• canary comparison,
• and auditability.

18. Treat Policy Configuration as an Artifact

Do not hide important runtime behavior in scattered constants.

A policy artifact might look like:

model_router:
  version: router-v7
  cheap_model: local-qwen
  strong_model: frontier
  failure_threshold: 0.31
  abstain_margin: 0.05

search_policy:
  version: search-v4
  default_width: 1
  uncertain_width: 3
  maximum_width: 5
  min_expected_gain: 0.04

critic_policy:
  version: critic-v2
  invoke_when_score_below: 0.72

stopping_policy:
  version: stop-v5
  require_verifier_pass: true
  max_unknown_retries: 1

Now architecture changes are explicit and reviewable.

19. Policy Diffing

Suppose a new production version suddenly costs 35% more.

Without policy lineage, debugging may take hours.

With policy artifacts:

-search.maximum_width: 3
+search.maximum_width: 5

-model_router.failure_threshold: 0.38
+model_router.failure_threshold: 0.27

The likely cause is immediately visible.

Policy configuration should be diffable for the same reason code is diffable.

20. Do Not Optimize Everything Simultaneously

If you simultaneously change:

• model router,
• search budget,
• critic invocation,
• escalation threshold,
• stopping policy,

and performance improves, you do not know why.

Worse, one change may be helping while another is hurting.

Prefer controlled experiments.

baseline
   ↓
change router only
   ↓
measure
   ↓
change search policy only
   ↓
measure

This is slower than architectural enthusiasm.

It is much faster than debugging a giant adaptive system later.

21. Feature Ablation

A learned policy can become more complicated than necessary.

Suppose the router uses:

17 features

Remove them one at a time.

If removing prompt_length changes nothing, delete it.

If removing verifier_available causes a major regression, keep it.

The same evidence-first principle applies to features.

feature exists
    ↓
measurable contribution?
    ├── yes -> keep
    └── no  -> delete

22. Policy Complexity Is a Cost

A more complex controller adds:

• latency,
• training burden,
• monitoring burden,
• calibration drift,
• more feature dependencies,
• harder replay,
• harder rollback,
• harder explanation.

So policy complexity belongs in the cost function.

A tiny decision tree that achieves 96.5% verified success may be preferable to a neural controller achieving 96.7%.

The difference may not earn the operational burden.

23. Example: Coding Agent Model Router

Imagine a coding agent has two models.

small model
strong model

Features:

files likely affected
presence of tests
historical task-class failure rate
stack depth
repo size
whether task touches migrations
whether task touches authentication

A simple policy could be:

def route_code_task(
    files_estimate: int,
    tests_present: bool,
    touches_migrations: bool,
    touches_auth: bool,
) -> str:
    if touches_auth or touches_migrations:
        return "strong"

    if files_estimate > 5:
        return "strong"

    if not tests_present:
        return "strong"

    return "small"

After enough verified trajectories, maybe a logistic model improves cost without reducing success.

Great.

But the explicit rule remains the baseline.

24. Example: Research Agent Search Budget

A research system may need more search when sources disagree.

Features:

number of independent sources
source disagreement
claim importance
source freshness
primary-source availability

Policy:

low disagreement -> stop
moderate disagreement -> retrieve 3 more sources
high disagreement + high-impact claim -> escalate research depth

The goal is not maximal retrieval.

The goal is enough evidence to reach the required confidence.

25. Example: DevOps Escalation Policy

A DevOps agent may begin in read-only diagnostic mode.

Possible stages:

local diagnostics
   ↓
read-only cluster inspection
   ↓
stronger model analysis
   ↓
human approval
   ↓
mutation

The adaptive policy may optimize when to request more diagnostics or a stronger model.

It should not optimize away the human approval boundary for dangerous mutations.

The safety envelope remains fixed.

26. Example: Browser Agent Retry Policy

A browser agent fails to submit a form.

Possible causes:

DOM changed
field validation failed
session expired
wrong selector
network timeout

A naive policy retries the same action.

A better policy uses the failure class.

network timeout -> bounded retry
selector missing -> re-observe DOM
validation error -> inspect field state
session expired -> re-auth flow

This is policy optimization through failure-specific control, not generic persistence.

27. Example: Mixture-of-Agents Runtime

A mixture-of-agents system may choose among:

direct answer
specialist model
Best-of-N
critic/revision
Tree of Thoughts
MCTS
multi-agent debate
frontier escalation

Do not ask one monolithic controller to choose among everything from day one.

Use staged routing.

For example:

task
 ↓
cheap difficulty classifier
 ├─ easy -> direct
 ├─ medium -> specialist / Best-of-N
 └─ hard -> advanced path
               ↓
          search-needed classifier
          ├─ no -> strong model
          └─ yes -> search policy

Hierarchical control keeps each decision narrow.

28. Policy Cascades

A useful production pattern is a cascade.

cheap path
   ↓ fail/uncertain
specialist path
   ↓ fail/uncertain
bounded search
   ↓ fail/uncertain
strong model
   ↓ fail/uncertain
UNKNOWN / human review

Each stage should have:

• an entry condition,
• a cost,
• an expected rescue probability,
• an exit condition,
• and a verifier.

The policy decides whether expected rescue justifies the next stage.

29. Measure Rescue Rate

For an escalation stage:

rescue_rate =
    failures_before_stage_that_become_PASS
    /
    stage_invocations

But rescue rate alone is not enough.

Also measure:

cost per rescue
latency per rescue
false-success change
UNKNOWN reduction

A stage that rescues 5% of tasks but adds 10x latency to all tasks may not be worth it.

30. Counterfactual Policy Questions

Trajectory observability lets us ask questions such as:

What if we had not escalated?
What if beam width had been 3 instead of 5?
What if the critic had been skipped?
What if this task had used specialist B?
What if we had stopped after verifier stage 1?

These are much more useful than:

Did the final run pass?

They isolate the value of a policy decision.

31. Policy Promotion Pipeline

A candidate policy should pass several stages.

candidate
   ↓
offline replay
   ↓
held-out benchmark
   ↓
shadow mode
   ↓
small canary
   ↓
production

At every stage compare against the current production policy.

A policy is not promoted because it looks clever.

It is promoted because it improves the required metrics.

32. Shadow Mode

In shadow mode the candidate policy makes decisions but does not control production behavior.

For each real request:

production policy -> actual action
candidate policy  -> shadow action

Now we can inspect disagreement.

route disagreement rate
search-budget disagreement
escalation disagreement
predicted cost delta
predicted success delta

High disagreement deserves investigation before canary deployment.

33. Canary Policy Deployment

After shadow validation:

95% production policy
5% candidate policy

Measure:

• verified success,
• false-success rate,
• UNKNOWN rate,
• cost,
• p95 latency,
• policy-specific failure modes.

Promotion should require explicit gates.

For example:

verified_success >= baseline - 0.2%
cost <= baseline - 8%
false_success <= baseline
p95_latency <= baseline + 5%

The exact numbers depend on the application.

The important point is that the promotion rule is explicit.

34. Rollback Must Be Boring

If a new policy causes regressions, rollback should be trivial.

active_policy = router-v8
        ↓ regression
active_policy = router-v7

Do not require retraining.

Do not require reconstructing old prompts.

Do not require reverse-engineering a controller LLM.

A reversible system learns faster because experiments are less dangerous.

35. Policy Drift

Even a good policy may become wrong over time.

Reasons include:

• model upgrades,
• tool changes,
• task-distribution shifts,
• new repositories,
• verifier changes,
• price changes,
• latency changes,
• new experts.

Monitor:

route distribution
expert utilization
calibration error
cost per verified success
policy disagreement
success by task class

A threshold calibrated six months ago may no longer be appropriate.

36. Model Upgrades Can Break the Router

Suppose the cheap model improves dramatically.

The old router may still send 40% of tasks to the expensive model.

Nothing is technically broken.

But the policy is now wasteful.

This is why model version belongs in policy evaluation.

policy effectiveness = f(policy, model versions, task distribution)

Policies cannot be evaluated independently from the capabilities they route between.

37. Price Changes Can Change the Optimal Policy

Suppose the strong model becomes 80% cheaper.

The reliability frontier may not change.

The economically optimal routing threshold can.

Agent architecture is partly an economics problem.

The optimal policy is not static when resource prices change.

38. Deterministic Policy Beats Learned Policy More Often Than People Expect

Some tasks have clear rules.

For example:

if production mutation -> require human approval
if no verifier available -> strong model
if migration file touched -> run migration verifier
if tests fail -> do not report PASS

These should remain deterministic.

Do not replace known invariants with statistical prediction.

Use learning where the boundary is genuinely uncertain.

39. Keep the Learned Surface Small

A strong architecture often looks like:

deterministic safety rules
          ↓
deterministic workflow structure
          ↓
small learned routing/search policies
          ↓
LLMs used for generative reasoning
          ↓
external verification

Not:

LLM decides everything

The runtime should absorb what can be made explicit.

The model should handle the parts that actually benefit from generative reasoning.

40. The Policy Should Be Explainable in One Record

When the runtime makes a decision, log something like:

{
  "decision": "escalate_model",
  "policy_version": "escalation-v4",
  "features": {
    "cheap_failure_probability": 0.43,
    "verifier_unknown": true,
    "retries": 1
  },
  "threshold": 0.31,
  "chosen_action": "frontier_model",
  "alternatives": ["retry_cheap", "stop_unknown"],
  "expected_extra_cost": 0.17
}

You do not need hidden reasoning.

You need operational evidence.

41. Build a Small Policy Runtime

Here is a compact provider-agnostic structure.

from dataclasses import dataclass
from typing import Any, Mapping


@dataclass(frozen=True)
class PolicyDecision:
    policy_name: str
    policy_version: str
    action: str
    reason: str
    features: Mapping[str, Any]
    score: float | None = None


class Policy:
    name: str
    version: str

    def decide(self, features: Mapping[str, Any]) -> PolicyDecision:
        raise NotImplementedError


class ThresholdEscalationPolicy(Policy):
    name = "escalation"

    def __init__(self, version: str, threshold: float):
        self.version = version
        self.threshold = threshold

    def decide(self, features: Mapping[str, Any]) -> PolicyDecision:
        p_fail = float(features["cheap_failure_probability"])

        if p_fail > self.threshold:
            action = "strong_model"
            reason = f"{p_fail:.3f} > {self.threshold:.3f}"
        else:
            action = "cheap_model"
            reason = f"{p_fail:.3f} <= {self.threshold:.3f}"

        return PolicyDecision(
            policy_name=self.name,
            policy_version=self.version,
            action=action,
            reason=reason,
            features=dict(features),
            score=p_fail,
        )

The point is not the code.

The point is that policy decisions become explicit objects.

They can be:

• logged,
• replayed,
• tested,
• benchmarked,
• diffed,
• and rolled back.

42. Test Policies Like Ordinary Software

A threshold policy should have unit tests.

def test_escalates_above_threshold():
    policy = ThresholdEscalationPolicy("v1", threshold=0.3)

    d = policy.decide({"cheap_failure_probability": 0.7})

    assert d.action == "strong_model"


def test_stays_cheap_below_threshold():
    policy = ThresholdEscalationPolicy("v1", threshold=0.3)

    d = policy.decide({"cheap_failure_probability": 0.1})

    assert d.action == "cheap_model"

Then add dataset-level tests:

verified success >= target
cost <= budget
false-success <= limit

A learned policy does not exempt the runtime from ordinary software engineering.

43. Test Policy Invariants

Useful invariants include:

never mutate production without authorization
never claim PASS without required verifier evidence
never exceed hard cost budget
never route outside allowed model set
never use stale policy artifact
never mix tenant feature data

These are stronger than statistical performance metrics.

They must hold on every run.

44. Monitor Policy Disagreement

If a candidate and production policy agree 99.8% of the time, the candidate probably cannot change much.

If they disagree 45% of the time, deployment risk is high.

Disagreement rate is therefore useful during policy development.

But disagreement alone is not good or bad.

We need to know whether disagreement occurs on the right tasks.

45. Measure Decision-Level Value

For each policy action, track outcomes.

Example:

Decision: invoke critic

invocations:          10,000
wrong -> correct:      1,420
correct -> wrong:        310
unchanged wrong:       2,100
unchanged correct:     6,170

Net correction:

1,420 - 310 = +1,110

Now include cost.

If critic calls are extremely expensive, the policy may still need tighter targeting.

46. Do Not Confuse Correlation With Policy Value

Suppose difficult tasks are more likely to use MCTS.

Those tasks also have lower success rates.

A naive analysis could conclude:

MCTS reduces success

That may be false.

The system routes MCTS to harder problems.

This is selection bias.

Use:

• matched task groups,
• counterfactual replay,
• randomized exploration where safe,
• shadow policies,
• controlled benchmark strata.

Trajectory data is powerful, but causal attribution still matters.

47. Controlled Exploration

If a policy never tries an alternative, it cannot learn whether the alternative improved.

For low-risk decisions we can allocate small exploration traffic.

For example:

95% current best route
5% safe alternative

But exploration must respect safety and cost limits.

Do not randomly explore dangerous production mutations.

Use exploration for decisions such as:

• model routing,
• search width,
• critic invocation,
• retrieval depth,
• safe diagnostic sequences.

48. Exploration Has a Budget Too

Track:

exploration cost
exploration success delta
new information gained
regressions caused

Exploration is not free just because it helps learning.

49. When Should You Use a Learned Controller?

A learned policy becomes more attractive when:

• the decision boundary is genuinely fuzzy,
• many verified trajectories exist,
• features are stable,
• the action set is narrow,
• counterfactual evaluation is possible,
• mistakes are observable,
• rollback is easy.

It is less attractive when:

• data is sparse,
• safety depends on the decision,
• the environment changes constantly,
• outcomes are weakly verified,
• the action space is huge,
• the learned policy is harder to understand than the problem.

50. A Useful Escalation Ladder for Policy Complexity

Start here:

hard-coded rule

If insufficient:

threshold table

Then:

small decision tree

Then:

logistic regression / calibrated classifier

Then perhaps:

gradient-boosted model / small neural policy

Only then ask whether an LLM controller is justified.

This is the same architecture principle we have used throughout the series:

Complexity must earn its place.

51. The Agent Control Plane

At this point we can see the runtime as two layers.

Execution plane

models
tools
memory
search
critics
verifiers

Control plane

routing policy
search policy
escalation policy
critic policy
stopping policy
budget policy

The control plane decides which execution-plane capabilities are used.

This is a useful architectural boundary.

It allows execution mechanisms to improve without rewriting the orchestration logic from scratch.

52. Policy Optimization Can Simplify the Execution Plane

Suppose the telemetry shows:

critic invoked 30% of runs
net verified improvement: 0.2%
latency increase: 18%

Delete the critic.

Suppose MCTS is used on 12% of tasks but only beats beam search on one narrow category.

Route only that category to MCTS.

Suppose the frontier model is called on 45% of tasks but the cheap model now performs equally well on half of them.

Raise the escalation threshold.

Optimization should often produce:

less architecture

not more.

53. The Complete Evidence Loop

We now have a closed system.

production tasks
      ↓
agent runtime
      ↓
decision traces
      ↓
external verification
      ↓
trajectory dataset
      ↓
attribution
      ↓
candidate policy
      ↓
offline replay
      ↓
compute-matched benchmark
      ↓
shadow
      ↓
canary
      ↓
promotion / rollback
      ↓
production tasks

That loop is much more important than any individual agent technique.

It gives us a way to improve the runtime without guessing.

54. Ten Rules for Agent Policy Optimization

If you remember only ten things, remember these.

1. Optimize decisions, not architecture aesthetics.
2. Use external verification as the outcome signal.
3. Optimize cost and latency subject to reliability constraints.
4. Start with explicit rules and thresholds.
5. Optimize routing, search, critics, escalation and stopping separately.
6. Calibrate probabilities before trusting thresholds.
7. Keep safety invariants outside the learned policy.
8. Version every policy and feature schema.
9. Promote through replay, shadow and canary stages.
10. Delete mechanisms that fail to earn their cost.

55. Where We Are Now

The series began with advanced orchestration techniques.

We then built toward something more important.

A disciplined agent runtime.

We now have:

advanced mechanisms
        ↓
architecture selection
        ↓
compute-matched benchmarking
        ↓
trajectory observability
        ↓
verified trajectory learning
        ↓
policy optimization

The result is not an agent that endlessly adds more intelligence-shaped machinery.

It is a system that can answer:

What failed?
Why did it fail?
Which mechanism was supposed to fix it?
Did that mechanism actually help?
How much did it cost?
Can we route it more selectively?
Can we simplify the architecture?

That is the difference between an impressive demo and an engineering system.

The next stage is to go one level deeper into the control plane itself: how to design budget allocation as a first-class resource scheduler across models, search branches, tools and verifiers rather than relying on fixed limits everywhere.