# Advanced Agents From First Principles 14: Can Your Agent Learn From Its Own Trajectories Without Learning the Wrong Lessons?
# Book: Advanced Agents From First Principles
# Chapter URL: https://programmer.ie/books/advanced-agents-from-first-principles/14-chapter/
# Prose fingerprint (sha256, normalized): 286eea8bcfe4592ae1c1a596437f42ffe92205eb1ebf2948db1a92ca45f48543
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

An advanced agent now leaves behind something extremely valuable:
evidence.
Not merely chat history.
Not merely model outputs.
Not merely traces.
A sufficiently instrumented system can record:
what state it was in, what alternatives it considered, which route it selected, what branches it pruned, which model or specialist it escalated to, which tools it called, which critic changed the answer, what verification evidence was produced, how much compute was spent, and whether the final result actually passed. That immediately suggests a tempting idea:
Why not let the agent learn from its own runs?
The answer is: you probably should.
But this is also one of the easiest ways to make an agent system quietly worse.
A successful trajectory does not prove that every decision inside it was good.
A failed trajectory does not prove that every decision inside it was bad.
A model can reach the correct answer despite a terrible plan.
A router can choose the wrong expert and still succeed because the expert is unusually strong.
A search policy can prune the best branch and still recover through another branch.
A critic can damage a correct answer while a later verifier catches the regression.
A cheap route can fail only because the task itself was impossible.
And a very expensive route can succeed while wasting 90% of its compute.
So the central rule of this post is:
Do not train from outcomes alone. Learn from verified trajectories with explicit attribution.
That distinction is the difference between adaptation and self-reinforcing noise.
The Dangerous Shortcut Imagine a production agent that handles coding tasks.
It records this run:
task ↓ router chooses frontier model ↓ planner generates 8-step plan ↓ Tree of Thoughts explores 16 branches ↓ critic revises candidate ↓ verifier runs tests ↓ PASS A naive learning system might conclude:
this task class -> frontier model -> deep search -> critic But perhaps the task was trivial.
Perhaps the same patch would have passed with:
small model ↓ one tool call ↓ unit tests ↓ PASS The first trajectory is successful.
It is not necessarily good.
This is why agent learning cannot simply optimize:
successful trajectory = reinforce failed trajectory = suppress That rule confuses outcome with causal contribution.
What Should an Agent Actually Learn? The useful target is usually not:
Learn how to answer this exact task.
It is:
Learn which runtime decisions improve verified outcomes for this class of state.
Those decisions can include:
routing search width search depth model choice critic invocation memory retrieval verification depth retry strategy escalation threshold tool ordering compute budget stopping policy Notice what is missing from that list.
The agent does not necessarily need to update the foundation model itself.
A huge amount of useful adaptation can happen in the runtime policy surrounding the model.
That is safer, cheaper, easier to inspect, and much easier to roll back.
The Learning Loop A practical architecture looks like this:
production task ↓ agent runtime ↓ trajectory trace ↓ external verification ↓ trajectory label ↓ offline attribution ↓ candidate policy change ↓ replay / benchmark ↓ shadow evaluation ↓ promote or reject ↓ new runtime policy The key word is offline.
Do not let one surprising production success immediately rewrite routing policy.
Do not let one failure immediately increase search depth globally.
Do not let the agent rewrite its own control logic because it thinks it learned something.
Instead:
collect evidence, derive a candidate change, benchmark it against the current policy, test it on held-out tasks, deploy it cautiously, keep rollback available. 1. Separate Outcome Labels From Decision Labels Suppose a run ends in:
PASS That tells you something important:
The final system state satisfied the verifier.
It does not tell you:
the route was optimal, the plan was useful, the critic helped, the search budget was justified, the memory retrieval was relevant, the expensive model was necessary, or the selected branch was the best one. We therefore need at least two levels of labels.
Run-level outcome from enum import Enum class Outcome(str, Enum): PASS = "pass" FAIL = "fail" UNKNOWN = "unknown" Decision-level attribution from dataclasses import dataclass @dataclass class DecisionAttribution: decision_id: str decision_type: str chosen_option: str alternatives: list[str] outcome: Outcome estimated_contribution: float | None confidence: float evidence_ids: list[str] The second object is harder to produce.
That is precisely why it is valuable.
2. Counterfactuals Are More Useful Than Raw Success Suppose the router chose expert A and the run passed.
Was A the right choice?
The strongest evidence would be:
same state ├─ expert A -> PASS, $0.05, 2.1 s ├─ expert B -> PASS, $0.02, 1.0 s └─ expert C -> FAIL, $0.01, 0.7 s Now you can say something meaningful.
A was capable.
B was better under the current cost objective.
C was insufficient.
Without the alternatives, you only know that A worked.
This is why trajectory observability from the previous post matters so much.
It gives us the information needed for counterfactual replay.
We can rerun selected decisions offline:
original production state ↓ replay router alternatives ↓ replay search policy alternatives ↓ replay critic on/off ↓ replay different budgets ↓ external verifier This gives the learning system evidence about what would have happened under another policy.
3. Learn From Pairwise Policy Comparisons Binary reinforcement is often too crude.
A more useful question is:
Given the same starting state, which policy produced the better verified outcome?
For example:
policy A verified = PASS cost = $0.11 latency = 8.2 s policy B verified = PASS cost = $0.03 latency = 2.9 s B dominates A.
Or:
policy A verified = PASS cost = $0.05 policy B verified = FAIL cost = $0.01 Now the choice depends on the value of success versus cost.
A simple utility function might be:
def utility(*, verified: bool, cost: float, latency: float) -> float: reward = 1.0 if verified else 0.0 return reward - 0.5 * cost - 0.01 * latency The exact weights are domain-specific.
The important point is conceptual:
Learn from comparisons under explicit objectives, not from success alone.
4. Do Not Learn From Unverified Success Suppose an agent says:
Done. Deployment succeeded.
If no external deployment state was checked, that trajectory should not enter the trusted learning set as a success.
Likewise:
model confidence = 0.97 critic says correct majority vote = yes None of those are equivalent to verification.
The training filter should be strict:
def eligible_for_learning(run) -> bool: return ( run.verification_status in {"pass", "fail"} and run.verifier_version is not None and run.state_id is not None and run.trace_complete ) UNKNOWN trajectories are still useful operationally.
But they should not be silently converted into success or failure labels.
5. Verification Quality Must Travel With the Trajectory A PASS from a weak verifier should not have the same weight as a PASS from a strong verifier.
Consider a coding agent.
Weak verification:
file exists Stronger verification:
file parses Stronger still:
unit tests pass And stronger again:
unit tests integration tests static checks requested behavior confirmed regression checks Store verifier metadata with the trajectory:
@dataclass class VerifiedOutcome: status: str verifier_id: str verifier_version: str coverage: float state_id: str evidence_ids: list[str] Learning signals should be weighted by verifier quality and coverage.
6. Avoid Reward Hacking at the Runtime Layer Reward hacking is not only a reinforcement-learning problem.
Agent runtimes can learn pathological shortcuts too.
Suppose your metric is:
maximize PASS rate The system may learn to route only easy tasks and return UNKNOWN for difficult ones.
Suppose your metric is:
minimize cost per task The router may stop escalating even when escalation is required.
Suppose your metric is:
minimize latency The agent may skip verification.
A more realistic objective is multi-dimensional:
verified success false-success rate UNKNOWN rate cost latency safety violations regressions coverage No single scalar should hide unacceptable failures.
You may optimize cost subject to reliability constraints:
minimize cost subject to: verified success >= 95% false-success <= 0.1% safety violations = 0 This is often safer than folding everything into one reward number.
7. The Agent Should Learn Different Things at Different Layers A useful production architecture does not have one giant adaptive policy.
Different subsystems learn from different evidence.
Router Learns:
task features -> expert choice Evaluate with:
routing accuracy, routing regret, missed escalation, unnecessary escalation, cost per verified success. Search policy Learns:
state features -> expansion / pruning / budget Evaluate with:
oracle survival, pruning regret, unique branch ratio, nodes per verified success, search depth used. Critic policy Learns:
state + candidate -> invoke critic? Evaluate with:
wrong -> correct correct -> correct wrong -> wrong correct -> wrong The last transition is especially important.
A critic that frequently changes correct -> wrong is actively harmful.
Escalation policy Learns:
uncertainty / failure signals -> stronger model? Evaluate with:
successful rescue rate, unnecessary escalation rate, marginal gain per escalation, cost of escalation. Verification policy Learns:
risk + state -> verification depth Evaluate with:
false-pass rate, false-fail rate, verification cost, coverage, detection of injected failures. This decomposition matters.
It prevents one successful run from blindly reinforcing every subsystem that happened to participate.
8. Credit Assignment Is the Hard Part Suppose this run passes:
router ↓ planner ↓ search ↓ critic ↓ verifier ↓ PASS Which component deserves credit?
Maybe:
the router selected the correct expert, the planner was irrelevant, search discovered the good branch, the critic nearly ruined it, the verifier caught the regression, recovery restored the earlier answer. If we only attach the final PASS to every component, we teach the critic the wrong lesson.
Better attribution comes from ablation and replay.
For example:
full system -> PASS without planner -> PASS without search -> FAIL without critic -> PASS without verifier -> false PASS Now the evidence is much clearer:
planner = no measurable contribution search = important critic = unnecessary verifier = critical This is one reason the previous two posts—benchmarking and observability—had to come before trajectory learning.
9. Build a Trusted Trajectory Dataset A production trace store is not automatically a training dataset.
The raw store may contain:
incomplete traces, transient failures, bad verifier versions, duplicate retries, test tasks, adversarial inputs, corrupted state identities, stale memory, human overrides, policy experiments, data from different product versions. Create a curated dataset explicitly.
A record might look like:
@dataclass class TrajectoryExample: trajectory_id: str task_class: str state_features: dict architecture_version: str policy_version: str outcome: VerifiedOutcome total_cost: float latency_ms: int decisions: list[DecisionAttribution] eligible: bool Then record why a trajectory was accepted:
verified state bound verifier version approved trace complete no policy experiment contamination no duplicate lineage no known data corruption Dataset curation is part of the learning system.
10. Preserve Time Ordering Agents that learn from previous runs create a subtle evaluation risk.
Imagine you collect tasks from January through June.
Then you randomly split trajectories into train and test sets.
The training set may contain patterns learned from May that influence evaluation on February tasks.
That may not match real deployment.
A safer evaluation is chronological:
past runs ↓ learn candidate policy ↓ future held-out runs For example:
train: Jan-Apr validation: May test: June This better measures:
Would the system actually have improved future behavior?
11. Prevent Cross-Task Leakage Some task families contain near duplicates.
Coding example:
fix bug in parser v1 fix bug in parser v2 fix same bug in forked repository Research example:
same question with slightly different wording Browser example:
same checkout flow across repeated runs If near-duplicate tasks appear in both training and evaluation, the adaptive policy can look much smarter than it really is.
Use grouping keys where possible:
repository issue family customer account website workflow type source document incident family Then split by group rather than individual trajectory.
12. Learn Routing Policies From Regret Suppose a router chose:
small_model and failed.
Offline replay shows:
small_model -> FAIL, $0.01 medium_model -> PASS, $0.03 frontier_model -> PASS, $0.15 The useful learning target is not:
small_model = bad It is:
for this state distribution, medium_model dominates small_model and frontier_model Routing regret can be approximated as:
chosen_utility = utility(chosen_result) best_utility = max(utility(r) for r in alternatives) regret = best_utility - chosen_utility High-regret regions are where routing policy improvement matters most.
13. Learn Search Budgets From Marginal Gain A search policy may use:
beam width = 8 depth = 6 But what if the winning branch appeared at depth 2 and all later expansion added no value?
Trajectory traces let us estimate:
verified gain after 1 node gain after 2 nodes gain after 4 nodes gain after 8 nodes ... Then the policy can learn where additional compute stops paying.
Conceptually:
if expected_gain(next_expansion) < expected_cost(next_expansion): stop_search() This does not require a neural controller.
A calibrated lookup table or small regression model may be enough.
14. Learn When Not to Invoke the Critic Critics are often treated as universally useful.
They are not.
Suppose production traces show:
critic invoked: 10,000 times wrong -> correct: 1,200 correct -> correct: 6,500 wrong -> wrong: 1,500 correct -> wrong: 800 The critic repairs many failures.
But it also creates 800 new ones.
Now stratify by task class.
Maybe:
code review: critic helpful SQL generation: critic neutral short factual extraction: critic harmful The learned policy becomes:
invoke critic only where expected net correction > cost + regression risk That is a much better architecture than:
always run critic 15. Learn Escalation Thresholds An adaptive system often has an escalation ladder:
cheap model ↓ if uncertain medium model ↓ if still unresolved frontier model The difficult question is where to place the thresholds.
Production trajectories provide features such as:
low scorer margin critic disagreement verification failure search stagnation high tool-error rate unknown memory retrieval branch disagreement The system can learn which combinations predict that escalation will help.
Example:
features = { "score_margin": 0.03, "critic_disagreement": 1, "search_stagnation": 0.8, "verification_failed": 1, } But again: evaluate the policy against fixed thresholds under equal average compute.
Adaptive does not automatically mean better.
16. Shadow Policies Before Promotion Never deploy a new learned policy merely because offline metrics improved.
Run it in shadow mode.
Production requests continue using policy A.
Policy B receives the same state and records what it would have chosen.
production state ├── policy A -> actual action └── policy B -> shadow decision Where safe, replay the shadow decision offline.
Then compare:
routing differences budget differences predicted cost predicted success regret Only after sufficient evidence should B receive live traffic.
17. Canary the Policy, Not Just the Model Agent architecture changes often happen outside the model.
A new router can cause regressions.
A new pruning rule can remove good branches.
A new critic threshold can increase false revisions.
A new memory policy can inject stale context.
Treat runtime policies like deployable software artifacts.
Version them:
router_v17 search_policy_v8 critic_policy_v4 verification_policy_v11 Then canary them:
1% traffic 5% 20% 50% 100% Monitor verified outcomes and rollback automatically when guardrails fail.
18. Never Let the Agent Rewrite Its Own Safety Boundary Some parts of the runtime should not be learned freely.
Examples:
authorization rules credential scope sandbox boundaries allowed tool set irreversible-action policies mandatory verification criteria tenant isolation secret filtering These are policy constraints, not optimization targets.
The adaptive layer may learn:
which safe tool to use when to escalate how much search to spend which expert to route to It should not learn:
whether to bypass authorization because doing so improved task completion A useful architecture is:
immutable safety envelope ↓ adaptive optimization layer ↓ agent runtime The optimization layer operates inside the safety boundary.
19. Detect Self-Reinforcing Failure Loops A dangerous adaptive loop looks like this:
router prefers expert A ↓ expert A gets more traffic ↓ more A trajectories collected ↓ training data dominated by A ↓ router becomes more confident in A ↓ expert B receives almost no traffic This is a feedback loop.
The system can become convinced that A is best simply because it stopped collecting evidence about B.
This is the classic exploration problem in a new costume.
Possible safeguards include:
minimum exploration traffic, randomized shadow evaluation, periodic counterfactual replay, uncertainty-aware routing, expert coverage metrics, entropy floors. Do not allow adaptive routing to eliminate the data needed to challenge itself.
20. Distinguish Learning From Memory This distinction matters.
Memory says:
for task X, event Y happened Learning says:
across many verified tasks, policy B tends to outperform policy A under conditions C Memory retrieves evidence.
Learning changes future behavior.
Confusing them creates brittle systems.
For example:
"Last time we used MCTS and passed" is memory.
It is not evidence that:
MCTS should now be used for every similar task. That second claim requires aggregation and controlled comparison.
21. Distinguish Learning From Caching Caching says:
same input + same relevant state -> reuse prior result Learning says:
change policy for future unseen states These should not be mixed.
If the task is identical and deterministic enough, caching may be the right solution.
Do not train a policy to rediscover something that could simply be reused exactly.
22. Prefer Small Policy Models First Suppose you want to learn routing.
You may not need another LLM.
A simple classifier can be enough:
features = [ task_length, tool_count, uncertainty, prior_failure_rate, verifier_strength, ] route = classifier.predict(features) Likewise, search budget can be estimated by a regression model.
Critic invocation can be a calibrated threshold.
Escalation can be a decision tree.
The policy should be as simple as the decision allows.
This keeps adaptation inspectable.
23. Version the Dataset Too If policy versioning matters, dataset versioning matters equally.
Record:
dataset_id collection window filters verifier versions architecture versions task families exclusions split method Otherwise a performance change may come from:
a different task distribution, a stronger verifier, duplicated trajectories, leaked future examples, or a different policy. You need to know which.
24. Measure Policy Stability A learned policy can improve average success while becoming unstable.
For example:
same task class run 1 -> small model run 2 -> frontier model run 3 -> MCTS run 4 -> small model Some stochasticity may be intentional.
But unexplained routing churn makes production systems hard to reason about.
Useful metrics include:
route flip rate budget variance critic invocation variance search-depth variance expert utilization drift policy entropy Unexpected changes should be investigated.
25. Detect Distribution Shift A policy learned from last month may stop working when the task mix changes.
Examples:
new repository language new browser UI new tool version new customer workflow new deployment infrastructure new model version Monitor features and outcomes over time.
If:
verified success ↓ routing regret ↑ UNKNOWN ↑ expert utilization shifts sharply then the learned policy may be outside its calibration region.
Fallback to a safer default or trigger re-evaluation.
26. Production Learning Needs Guardrail Metrics Do not promote a policy based on average success alone.
Monitor:
verified success false-success rate UNKNOWN rate cost per verified success p95 latency safety violations routing regret pruning regret critic regression rate unnecessary escalation missed escalation memory contamination A policy that improves one metric by destroying another should not silently win.
27. Coding Agent Example Suppose a coding agent performs thousands of repository tasks.
The runtime records:
repository features issue type selected model search depth critic usage tool calls test results verification outcome cost latency After enough runs, you may discover:
small bug fixes: local model + tests = best cross-module refactors: frontier model + planning = best ambiguous architecture tasks: frontier + bounded search = best critic: useful on refactors harmful on mechanical edits The runtime can learn this policy.
But exact repository state must still come from Git and the filesystem.
Do not replace current state with learned memory.
And final success should still come from tests and explicit acceptance criteria.
28. Research Agent Example A research system may learn:
simple factual lookup -> direct search + source verification multi-source comparison -> broader retrieval + synthesis contested claim -> adversarial source search + disagreement review high-stakes claim -> stronger verifier + source diversity requirement Useful trajectory features include:
source disagreement citation coverage claim count retrieval diversity failed query count source authority But the system should not learn:
"source X agreed with us before, so trust it more" without independent evidence.
That is how confirmation bias becomes architecture.
29. DevOps Agent Example DevOps is where unsafe adaptation becomes especially dangerous.
The agent may learn:
read-only diagnostics first then safe remediation then stronger escalation Useful learned decisions:
which diagnostic to run which expert to consult how long to search when to escalate to human review Non-learnable hard constraints might include:
never delete production data automatically never widen credentials never bypass approval always verify deployment state always preserve rollback path The adaptive layer optimizes inside that envelope.
30. Browser Agent Example Browser agents create many repeated trajectories.
They can learn:
which selectors are reliable when navigation is stuck when OCR/vision is required when to abandon a stale page state which checkout flows require confirmation But current DOM state should remain a live observation.
Do not let a learned policy assume that last week’s checkout page still exists.
Verified external state beats learned expectation.
31. Multi-Agent Systems Can Learn Team Composition A mixture-of-agents runtime may have:
planner coder researcher critic verifier small model frontier model search agent It does not need to invoke all of them.
Trajectory evidence can teach:
which specialists help which task classes which pairs duplicate each other which critics correlate in their errors which debate configurations actually correct mistakes This can lead to a smaller architecture over time.
That is a healthy sign.
Learning should remove unnecessary machinery as often as it adds it.
32. Learning Can Simplify the Agent This is one of the most important consequences of evidence-driven adaptation.
Suppose you discover:
planner contributes no gain on 70% of tasks critic hurts extraction tasks MCTS adds cost without gain on deterministic refactors frontier model rarely needed for routing The correct learned architecture may be:
fewer calls fewer roles less search more deterministic routing That is not regression.
That is optimization.
Advanced agents should become simpler where evidence permits.
33. A Minimal Offline Policy Learner Here is a deliberately simple example.
from dataclasses import dataclass from collections import defaultdict @dataclass class RouteResult: task_class: str route: str verified: bool cost: float def score(r: RouteResult) -> float: return (1.0 if r.verified else 0.0) - r.cost def learn_route_policy(results: list[RouteResult]) -> dict[str, str]: grouped = defaultdict(lambda: defaultdict(list)) for r in results: grouped[r.task_class][r.route].append(score(r)) policy = {} for task_class, routes in grouped.items(): means = { route: sum(values) / len(values) for route, values in routes.items() } policy[task_class] = max(means, key=means.get) return policy This is intentionally crude.
It demonstrates the architecture:
verified trajectories ↓ group by task condition ↓ compare routes ↓ learn candidate policy A real system should add:
confidence intervals, minimum sample sizes, held-out evaluation, time-based splits, safety constraints, regret analysis, drift monitoring. But the learning loop does not have to begin with a complicated neural controller.
34. Require Minimum Evidence Before Changing Policy Imagine route B wins after:
A: 2 runs B: 1 run That is not enough evidence.
Set promotion requirements:
minimum samples minimum effect size confidence interval excludes unacceptable regression no safety regression no false-success increase held-out improvement A policy change should have an evidence threshold just like any other engineering change.
35. Compare Against the Current Policy, Not an Imaginary Baseline The baseline is the production policy currently serving users.
Candidate B must beat A.
policy A: production policy B: candidate Do not compare B only against:
one-shot LLM if production already uses routing, memory and verification.
Incremental architecture changes require incremental comparisons.
36. Keep a Policy Lineage Every promoted policy should record:
parent policy training dataset benchmark run metrics promotion reason rollback target For example:
router_v18 parent: router_v17 dataset: trajectories_2026_07 benchmark: route_eval_44 improvement: -18% cost / same verified success promoted: 2026-08-09 Now the adaptive runtime has an engineering history.
You can answer:
Why does the agent route these tasks differently now?
That is far more useful than:
The system learned it somehow.
37. Do Not Train Directly on Every Failure Failures are not automatically lessons.
A failed run may be caused by:
network outage tool unavailable invalid test fixture external service failure bad verifier permission problem corrupted environment If you teach the policy from all failures indiscriminately, infrastructure noise becomes behavioral training data.
Classify failure causes first.
agent-caused external verifier-caused infrastructure unknown Only then decide how the trajectory should influence policy.
38. Unknown Causality Should Stay Unknown There will be runs where you know:
FAIL but you do not know why.
Do not invent attribution.
A useful decision record can say:
outcome = FAIL attribution = UNKNOWN That is scientifically cleaner than falsely blaming the router or planner.
Collect more evidence.
39. The Closed Improvement Loop We can now connect the last three posts.
production trajectories ↓ trajectory observability ↓ verified outcomes ↓ compute-matched benchmarking ↓ decision attribution ↓ candidate policy change ↓ held-out replay ↓ shadow deployment ↓ canary ↓ promotion ↓ new production trajectories That is a genuine adaptive agent runtime.
It does not blindly self-modify.
It accumulates evidence and changes policy only when the evidence supports the change.
40. The Most Important Rule An advanced agent that learns from its own history can become dramatically more efficient.
It can discover:
which tasks require frontier models, which tasks do not, where search helps, where search is wasteful, when critics repair errors, when critics create them, when escalation is worth its cost, which experts are redundant, and where deterministic software should replace the agent entirely. But the learning mechanism itself must remain subordinate to evidence.
The rule is:
A trajectory is evidence about what happened. It is not automatically evidence about what should happen next.
To cross that boundary, you need verification, attribution, comparison, held-out evaluation, and controlled promotion.
That is how an agent learns without simply becoming more confident in its own history.