Your Infrastructure Is Healthy. Why Is the Agent Getting Worse?

Your dashboards are green.

The model endpoint is responding.

The browser workers are alive.

The database is healthy.

The queue is draining.

The verifier service is up.

Latency has not exploded.

There are no obvious exceptions.

And yet the agent is getting worse.

It fixes fewer bugs.

It retrieves weaker evidence.

It escalates to expensive models more often.

It chooses the wrong tools more frequently.

Its verifier passes more incorrect results.

Its average cost rises even though task difficulty has not obviously changed.

Its success rate falls slowly enough that nobody notices for a week.

This is one of the most dangerous failure modes in production agent systems:

The platform is technically healthy while the behavior is quietly drifting.

The previous post dealt with dependency failure.

A failing model endpoint can trip a circuit breaker.

A saturated browser pool can trigger backpressure.

A broken retrieval service can enter degraded mode.

Those failures are visible.

Behavioral drift is harder because nothing has to crash.

The system can keep returning 200 OK while becoming less useful.

This post is about detecting that kind of failure.

The core distinction: infrastructure health is not behavioral health

A production agent has at least two very different kinds of health.

infrastructure health
    ↓
can the system execute?

behavioral health
    ↓
is the system still making good decisions?

Infrastructure health includes things such as:

• uptime,
• request success rate,
• CPU/GPU saturation,
• queue depth,
• database latency,
• API availability,
• browser-worker health,
• memory pressure,
• timeout rates.

Behavioral health includes things such as:

• verified task success,
• false-success rate,
• route accuracy,
• escalation quality,
• search efficiency,
• critic net-correction rate,
• verifier precision,
• retrieval usefulness,
• tool-selection accuracy,
• cost per verified success.

A system can be healthy on the first list and unhealthy on the second.

That is the problem.

What can drift?

Almost every adaptive or replaceable component in an advanced agent system can drift.

Model drift

A provider silently changes a model version.

Or you intentionally upgrade from one model to another.

The new model may be better overall while being worse on the tasks that matter to you.

Possible symptoms:

• more tool hallucinations,
• longer answers,
• worse structured output,
• weaker code edits,
• different refusal behavior,
• different calibration,
• different tendency to call tools,
• different susceptibility to critic feedback.

Prompt drift

A prompt edit looks harmless.

It changes one instruction.

That instruction alters routing behavior across thousands of tasks.

Prompt drift can come from:

• a manual edit,
• template refactoring,
• context-window changes,
• reordered instructions,
• newly injected memories,
• additional tool descriptions,
• changed examples.

Router drift

A routing policy may become miscalibrated as the task distribution changes.

Perhaps the router was trained when 80% of work was simple.

Now 50% is repository-scale engineering work.

The policy itself may be unchanged.

The environment around it changed.

Retrieval drift

The retrieval corpus changes constantly.

Documents are added.

Documents become stale.

Embedding models change.

Chunking strategies change.

Indexes are rebuilt.

A retrieval service can be perfectly available while returning progressively less useful evidence.

Verifier drift

This one is especially dangerous.

A verifier may continue returning answers while its acceptance boundary changes.

Examples:

• tests become weaker,
• benchmark fixtures become stale,
• a judge model becomes more permissive,
• a schema validator stops checking a field,
• an external API changes semantics,
• a deployment check examines the wrong environment.

The worst case is not verifier failure.

The worst case is a verifier that keeps saying PASS when it should say FAIL.

Policy drift

The control plane itself changes.

A new scheduler might:

• search more aggressively,
• escalate sooner,
• invoke critics more often,
• stop too early,
• select cheaper models too frequently,
• starve verification under pressure.

Data-distribution drift

Nothing in the system changes.

The tasks change.

This is often the real reason production metrics move.

For example:

last month
70% small bug fixes
20% refactors
10% repository migrations

this month
20% small bug fixes
30% refactors
50% repository migrations

A lower success rate may not mean the agent regressed.

The workload became harder.

This is why global averages are dangerous.

You need behavioral versioning

If you cannot answer what changed, you cannot attribute drift.

Every production trajectory should include versions for the components that can influence behavior.

For example:

run_id
architecture_version
model_version
prompt_version
router_policy_version
search_policy_version
budget_policy_version
critic_version
retrieval_index_version
embedding_version
verifier_version
tool_registry_version
memory_schema_version

This turns a vague question:

Why did success fall this week?

into something testable:

Did verified success fall after router_policy_v17 replaced router_policy_v16 for repository-scale coding tasks?

That is a much better question.

Never monitor only one number

Suppose your global success rate changes from:

82% → 78%

Is that meaningful?

Maybe.

But you do not know enough yet.

The change could come from:

• more difficult tasks,
• one bad tenant,
• one broken tool,
• one new model version,
• one verifier regression,
• more UNKNOWN outcomes being correctly reported,
• a real quality regression.

The correct monitoring unit is usually a vector.

For example:

verified success
false success
false failure
UNKNOWN rate
cost / verified success
p50 latency
p95 latency
model calls / run
tool calls / run
search nodes / run
escalation rate
critic invocation rate
verification coverage

Then slice it.

Slice by cohort before declaring drift

A global average can hide almost anything.

At minimum, segment by dimensions such as:

task class
risk class
tenant
repository size
model version
router version
tool family
verifier version
workload source
complexity bucket

Imagine this result:

simple coding tasks       91% → 91%
repository refactors      79% → 62%
research tasks            84% → 83%
DevOps diagnostics        88% → 87%

The global average might move only a few points.

But one cohort collapsed.

That is the signal you care about.

Use a baseline hierarchy

There is no single correct baseline.

You usually need several.

Previous-version baseline

Compare the candidate against the version it replaces.

policy_v17 vs policy_v16

This is the strongest baseline for deciding whether a deployment should remain live.

Rolling production baseline

Compare recent behavior to a recent stable window.

For example:

last 6 hours
vs
previous 7 stable days

Useful for detecting operational shifts.

Long-term reference baseline

Keep a durable reference suite that does not change with production traffic.

This helps distinguish workload changes from system changes.

Deterministic baseline

Where applicable, compare against deterministic software or a simple workflow.

If an advanced agent slowly degrades until it performs worse than a deterministic baseline, the problem is architectural, not cosmetic.

Change-point detection is better than arbitrary thresholds

A rule like:

alert if success < 80%

is often too crude.

Suppose a mature system normally runs at 96%.

Dropping to 89% is catastrophic even though the threshold is not crossed.

Or suppose a difficult workload normally runs at 65%.

A static 80% threshold is meaningless.

What you really care about is change relative to expected behavior.

Conceptually:

observed metric
      ↓
expected distribution
      ↓
change detector
      ↓
possible drift

You can begin with simple methods:

• rolling mean,
• rolling median,
• exponentially weighted moving average,
• bootstrap confidence interval,
• paired task comparison.

You do not need sophisticated statistics to start.

The key is to compare like with like.

Pairwise comparisons are extremely powerful

If you have a stable evaluation set, compare old and new systems on the same tasks.

For each task:

old policy → PASS
new policy → PASS

old policy → FAIL
new policy → PASS

old policy → PASS
new policy → FAIL

old policy → FAIL
new policy → FAIL

The most important cell is:

old PASS → new FAIL

That is direct regression evidence.

The reverse cell:

old FAIL → new PASS

is direct improvement evidence.

Aggregate metrics can hide this transition structure.

Shadow evaluation catches drift before deployment

A candidate component does not need to control production immediately.

Run it in shadow mode.

production request
      ↓
production policy → real action
      ↓
shadow policy → simulated decision

The shadow policy can record:

• chosen route,
• selected model,
• proposed search width,
• escalation decision,
• predicted verifier outcome,
• expected cost.

Then compare it to what actually happened.

Shadow mode is especially useful for:

• routers,
• search schedulers,
• escalation policies,
• critic policies,
• model selection,
• retrieval ranking.

It is less useful when the counterfactual outcome cannot be observed safely.

Do not invent outcomes you did not observe.

Canary deployments reduce blast radius

After offline and shadow evaluation, expose the candidate to a small traffic slice.

For example:

1%
↓
5%
↓
20%
↓
50%
↓
100%

Promotion should depend on evidence.

Not elapsed time.

A candidate should advance only if it satisfies explicit gates.

For example:

verified success degradation <= 0.5 percentage points
false-success rate does not increase
cost per verified success <= +5%
p95 latency <= +10%
no critical cohort regression
no verifier coverage reduction

Different systems will choose different numbers.

The important part is making the gates explicit before deployment.

Automatic rollback should be conservative

Automatic rollback is powerful.

It can also flap or amplify noise.

A safer rollback policy usually requires several conditions.

For example:

if
  regression confidence is high
  AND sample size is sufficient
  AND regression affects verified outcomes
  AND a known-good version exists
then
  rollback

Do not automatically roll back because one metric twitched.

Use hysteresis.

For example:

promotion threshold: +2% verified success
rollback threshold:   -1% verified success

Different thresholds reduce oscillation.

Roll back behavior, not history

Suppose a router policy is bad.

You roll it back.

That does not mean deleting evidence collected under the bad policy.

Keep the trajectories.

They are useful evidence.

The rollback should restore execution policy while preserving historical lineage.

policy_v17 deployed
      ↓
regression detected
      ↓
rollback to policy_v16
      ↓
retain v17 trajectories for analysis

This matters for learning.

A failed policy version may contain some of your best training examples.

Model upgrades deserve full architecture evaluation

A common mistake is treating a model upgrade as a drop-in infrastructure change.

It is not.

A new model changes the behavior of the entire agent loop.

It may affect:

• tool routing,
• structured output,
• planning,
• candidate diversity,
• critic behavior,
• verifier behavior,
• escalation frequency,
• context usage,
• token cost,
• latency.

Therefore evaluate the model inside the architecture.

Not only with standalone benchmark prompts.

For example:

old model + current router + current verifier
vs
new model + current router + current verifier

Then test whether dependent policies need recalibration.

A better model may require a different routing threshold.

Verifier drift requires special treatment

The verifier sits near the truth boundary.

If it drifts, your monitoring system can become self-deceptive.

Imagine:

real quality ↓
verifier permissiveness ↑
reported PASS rate stable

Your dashboards may look fine.

This is why verifier quality needs independent checks.

Use things such as:

• deterministic acceptance tests,
• gold cases,
• adversarial failure injection,
• manual audit samples,
• cross-verifier disagreement,
• immutable reference fixtures.

The verifier must itself be monitored.

Retrieval drift is often invisible in aggregate metrics

Suppose your retrieval index grows from 100,000 documents to 2 million.

Availability is still 99.99%.

Latency is acceptable.

But relevant evidence is now buried.

Metrics should include:

precision@k
recall@k
source freshness
source authority
retrieval diversity
stale-result rate
useful-retrieval rate

And more importantly:

retrieval → decision change
retrieval → verified outcome improvement

Retrieval quality should be judged by downstream usefulness, not only similarity score.

Router drift is a classification problem

The router chooses among alternatives.

For example:

cheap model
frontier model
search
retrieval
critic
human escalation

Drift appears when the relationship between router features and the best action changes.

Monitor:

• routing accuracy,
• routing regret,
• unnecessary escalation,
• missed escalation,
• route-flip rate,
• expert utilization,
• cost-weighted routing regret.

A router can remain internally confident while becoming wrong more often.

Confidence is not enough.

Policy drift can come from environment drift

Suppose your scheduler learned:

search width = 4

because that was optimal for the old model.

A newer model may generate more diverse candidates.

The same search width might now be wasteful.

Or a cheaper model may make larger search economically attractive.

Therefore policy quality is conditional on the environment.

A useful policy record includes:

policy_version
trained_on_model_version
trained_on_prompt_version
trained_on_tool_registry
trained_on_task_distribution
trained_on_price_table

This makes invalidation possible.

Drift detectors should reason about causality carefully

Suppose success falls immediately after a model upgrade.

That is suggestive.

It is not proof.

Maybe the task mix changed at the same time.

Maybe a verifier changed.

Maybe a repository migration produced unusually difficult tasks.

This is why trajectory lineage matters.

Use cohort comparison and version attribution.

For example:

same task class
same verifier
same retrieval index
same tool registry
old model vs new model

The closer you can get to matched conditions, the stronger the inference.

Build a drift event, not just an alert

A useful drift system should produce a structured artifact.

For example:

from dataclasses import dataclass
from typing import Optional


@dataclass(frozen=True)
class DriftEvent:
    metric: str
    cohort: str
    baseline_version: str
    candidate_version: str
    baseline_value: float
    observed_value: float
    delta: float
    confidence: float
    sample_size: int
    suspected_component: Optional[str]
    action: str

Example:

metric: verified_success
cohort: repository_refactor
baseline_version: model-2026-07
candidate_version: model-2026-08
baseline_value: 0.81
observed_value: 0.68
delta: -0.13
sample_size: 422
suspected_component: generator_model
action: rollback_candidate_model

Now drift becomes operational data.

Version the rollback decision too

Rollback logic is policy.

Policy should be versioned.

For example:

rollback-policy-v4

Why?

Because a bad rollback policy can be as dangerous as a bad router.

You should be able to answer:

• which metric triggered rollback?
• which threshold applied?
• what sample size existed?
• which cohorts were affected?
• what stable version was selected?
• who or what authorized the rollback?

Rollback must respect distributed execution

Step 20 introduced leases and fencing.

That matters here.

Suppose policy version 23 is rolled back while workers are still executing it.

Those workers may continue for minutes.

A platform should not assume rollback instantly removes old behavior.

Possible approaches include:

• let in-flight tasks finish,
• cancel low-risk work,
• fence mutation authority by policy epoch,
• revalidate before commit,
• require new tasks to use the restored policy.

For high-risk mutations, policy version can become part of the commit contract.

Rollback must respect platform scheduling too

Step 21 introduced the global scheduler.

Suppose a regression causes excessive frontier-model escalation.

That increases load.

The scheduler may interpret the load as traffic growth rather than behavioral drift.

The two control planes must exchange signals.

For example:

router drift
   ↓
frontier escalation ↑
   ↓
GPU pressure ↑
   ↓
platform scheduler constrained

Without behavioral telemetry, the platform sees only pressure.

With drift attribution, it can identify the cause.

Failure containment and drift detection solve different problems

Step 22:

dependency unhealthy
      ↓
contain failure

Step 23:

dependency healthy
behavior worse
      ↓
detect regression

You need both.

Circuit breakers do not catch silent semantic degradation.

Drift detectors do not replace circuit breakers.

Domain example: coding agent

Suppose a coding agent upgrades its generator model.

Infrastructure metrics look normal.

After three days:

compile success       unchanged
unit-test pass rate   -2%
regression pass rate  -11%
average patch size    +38%
critic invocation     +42%
cost / success        +31%

This suggests a behavioral change.

Slice by task type.

You discover:

small bug fixes      stable
API refactors        stable
cross-module fixes   severe regression

Now compare old and new models on matched cross-module tasks.

If the old model consistently wins, rollback is justified.

Domain example: research agent

The research agent still retrieves documents quickly.

But citations become less authoritative.

Possible drift signals:

primary-source rate ↓
source freshness ↓
claim coverage ↓
contradiction rate ↑
verification UNKNOWN ↑

The retrieval system is alive.

The evidence quality changed.

That is behavioral drift.

Domain example: browser agent

A browser agent still executes actions.

But a website changes its DOM.

The automation now reaches the wrong controls more often.

Signals:

selector mismatch ↑
recovery actions ↑
form correction rate ↑
postcondition failure ↑

A browser worker health check will not catch this.

The environment semantics drifted.

Domain example: data agent

A data agent processes a new upstream schema.

The pipeline does not crash.

But categorical values shift.

The agent begins selecting weaker transformations.

Useful drift signals include:

schema-change rate
validation-error distribution
repair strategy mix
manual override rate
post-transform invariant failures

Domain example: DevOps agent

A DevOps agent changes its incident-routing policy.

After deployment:

cheap diagnostics ↓
frontier escalation ↑
incident resolution time unchanged
cost ↑

That is not a reliability win.

It is a cost regression.

Behavioral drift includes economic behavior, not only correctness.

Drift metrics should include efficiency

A system can maintain success while degrading economically.

For example:

verified success = 90% → 90%
cost / verified success = $0.40 → $1.20

That is drift.

Likewise:

verified success stable
p95 latency doubles

That matters.

Monitor the Pareto surface, not only accuracy.

False success deserves the highest severity

Suppose:

reported PASS rate stable
false-success rate rises

This is worse than ordinary success degradation.

The system is becoming more confident in wrong outcomes.

Treat false-success drift as a high-severity event.

Especially for:

• code deployment,
• browser transactions,
• database mutation,
• incident remediation,
• financial or operational actions.

UNKNOWN can increase for good reasons

Suppose a verifier becomes stricter.

UNKNOWN rises.

Verified success falls slightly.

That may actually be an improvement if false successes fall significantly.

This is why metrics must be interpreted together.

Example:

before
PASS 90%
FAIL 8%
UNKNOWN 2%
false PASS 4%

after
PASS 84%
FAIL 9%
UNKNOWN 7%
false PASS 0.5%

The second system may be much safer.

Do not optimize one number blindly.

Protect against alert fatigue

If every small variation creates an alert, operators will ignore all of them.

Useful drift alerts should include:

what changed
where it changed
how large the change is
how confident we are
which component versions differ
what rollback candidate exists

Bad alert:

Agent success down 3%

Better alert:

Repository-refactor verified success fell 11.8 percentage points
following router-policy-v17 rollout.
Matched old-policy shadow comparison shows +9.4 pp recovery.
False-success rate unchanged.
Recommended action: rollback router-policy-v17.

That is actionable.

Drift detection itself can drift

There is another recursive problem.

Your monitoring policy may become stale.

Task distributions change.

Baseline windows become inappropriate.

Metrics lose meaning.

Therefore drift detection should also be versioned and periodically evaluated.

For example:

drift_detector_version
baseline_window
minimum_sample_size
rollback_threshold
cohort_schema
metric_definition_version

A compact production design

A useful minimal architecture looks like this:

production trajectories
        ↓
verified outcomes
        ↓
metric aggregation
        ↓
cohort slicing
        ↓
baseline comparison
        ↓
drift event
        ↓
shadow / replay confirmation
        ↓
rollback gate
        ↓
restore known-good version

Notice what is missing.

There is no LLM deciding whether drift exists.

You can add models later if they help explain anomalies.

The core detector can remain ordinary software.

Start simpler than you think

A first production system may need only:

versioned runs
verified outcomes
rolling metrics
cohort slicing
paired benchmark set
manual rollback

Then add:

shadow evaluation
canaries
automatic rollback
change-point detection
counterfactual replay

only when the operational evidence justifies them.

The same rule from the entire series still applies:

Do not add a mechanism because it sounds advanced. Add it because a measured failure requires it.

The final control loop

At this point the advanced-agent platform has become a closed engineering loop.

task
 ↓
admission
 ↓
routing
 ↓
budget allocation
 ↓
information gathering
 ↓
search / execution
 ↓
verification
 ↓
trajectory
 ↓
behavioral metrics
 ↓
drift detection
 ↓
rollback / policy update

Every major decision leaves evidence.

Every component is versioned.

Every adaptation is reversible.

Every claimed improvement is compared against a baseline.

And the system is allowed to say:

UNKNOWN

when reality cannot be established confidently.

That is a much stronger foundation than simply making the agent call a bigger model.

Practical checklist

Before calling an agent platform production-ready, ask:

1. Can we tell infrastructure failure from behavioral drift?
2. Are model, prompt, policy, retrieval and verifier versions attached to every trajectory?
3. Do we monitor verified success rather than self-reported success?
4. Do we slice metrics by task class and risk class?
5. Can we compare candidate versions against the exact version they replace?
6. Do we use shadow or paired evaluation before broad rollout?
7. Are canary promotion gates explicit?
8. Is rollback safe under in-flight distributed work?
9. Can verifier drift be detected independently?
10. Do we monitor cost and latency regressions as well as correctness?
11. Is false-success drift treated as high severity?
12. Can the system preserve historical evidence after rollback?
13. Are rollback decisions themselves versioned and observable?
14. Can we restore a known-good policy quickly?

If several answers are no, the platform may be operationally healthy while behavior deteriorates unnoticed.

The deeper lesson

Advanced agents are stochastic systems embedded inside ordinary software systems.

The stochastic part gets most of the attention.

The ordinary software around it determines whether the system is controllable.

Behavioral drift detection is one of those ordinary engineering mechanisms.

It does not make the model smarter.

It makes the platform less willing to become silently worse.

And that may be more valuable.

The next stage is release engineering for agent behavior: defining behavioral contracts, promotion gates, compatibility checks, migration plans, and versioned rollout policies so agent architecture changes can be shipped with the same discipline as database schemas or public APIs.