Is This Task Outside Your Agent’s Competence? Build Competence Envelopes and OOD Detection

Your agent can be confident and still have no business doing the task.

That sounds obvious.

It is not how many agent systems are built.

A task arrives.

The model understands the words.

The router finds a plausible expert.

The tools are available.

The agent produces a plan.

Nothing crashes.

So the system proceeds as though the task belongs to the class of problems the agent can reliably solve.

That assumption is dangerous.

A coding agent that performs well on ordinary Python repository maintenance may be asked to modify a safety-critical embedded controller.

A research agent trained and benchmarked on public web sources may be asked to reason from a sparse, adversarial set of internal documents.

A browser agent that reliably fills ordinary forms may encounter a financial transfer workflow with irreversible consequences.

A DevOps agent that handles routine deployments may be asked to recover a degraded distributed database during a regional outage.

In every case, the agent may be able to produce fluent output.

That is not the same thing as demonstrated competence.

Step 29 gave us authority boundaries and explicit human escalation.

The next question is harder:

How do we know whether the current task is inside the region where this agent has evidence-backed competence at all?

That is the problem of the competence envelope.

The core rule for this post is:

Autonomy should depend on demonstrated competence for the current task regime, not on the agent’s ability to produce an answer.

The Search Problem: “How Do I Know When an AI Agent Is Out of Distribution?”

Most production agent systems have a confidence problem.

They ask the model how confident it is.

Or they look at token probabilities.

Or they ask a critic whether the answer looks good.

Or they count how many candidate agents agree.

Those signals may sometimes be useful.

But none of them answers the question we actually care about:

Have we demonstrated that this system is reliable
on tasks sufficiently similar to this one,
under sufficiently similar operating conditions,
with sufficiently strong verification?

That is a different question.

A model can be highly confident on an unfamiliar task.

A critic can approve a failure mode it shares with the generator.

Five models can agree because they were trained on similar data.

A nearest-neighbor embedding can say a task is semantically similar while the operational risk is completely different.

The competence envelope therefore cannot be a single confidence score.

It has to be a structured evidence boundary.

Uncertainty Is Not the Same as Competence

Step 17 decomposed uncertainty into categories such as:

interpretation uncertainty
evidence uncertainty
route uncertainty
state uncertainty
tool uncertainty
candidate uncertainty
verification uncertainty

Those categories describe uncertainty inside a run.

Competence asks a different question:

Should this run be trusted to operate autonomously
in this task regime in the first place?

You can have:

low uncertainty + low competence evidence

For example:

Task:
    “Rotate the production signing key across all regions.”

Agent state:
    interpretation uncertainty: low
    evidence uncertainty: low
    tool uncertainty: low
    verification uncertainty: moderate

Historical evidence:
    zero production key-rotation cases
    zero cross-region rollback cases
    zero benchmark coverage for cryptographic key management

The agent may understand exactly what the request means.

It may even know technically correct procedures.

But the system has not demonstrated operational competence for that workload.

That should matter more than confidence.

Define a Competence Envelope

A competence envelope is the region of task and operating conditions for which you have enough evidence to grant a particular level of authority.

A useful envelope might include dimensions such as:

CompetenceEnvelope
├── task classes
├── domain classes
├── risk classes
├── environment classes
├── tool families
├── required verifier strength
├── repository / data scale ranges
├── dependency-health assumptions
├── latency / cost operating ranges
├── known failure exclusions
├── minimum evidence volume
└── allowed authority level

For a coding agent:

validated:
    Python services
    < 500k LOC
    unit + integration tests available
    ordinary CRUD / API changes
    no production credential mutation
    no cryptographic protocol changes
    repository has clean working tree

partially validated:
    mixed Python / TypeScript
    migration-heavy changes
    flaky integration tests

not validated:
    kernel code
    embedded firmware
    safety-critical control systems
    cryptographic primitives

The envelope is not a claim that the agent cannot solve tasks outside it.

It is a claim about what the system has evidence to trust.

That distinction matters.

Competence Must Be Authority-Specific

Do not ask:

Is the agent competent?

Ask:

For this task,
with this evidence,
what authority level has been justified?

Remember the authority classes from Step 29:

A0 — observe
A1 — propose
A2 — reversible execution
A3 — bounded external effect
A4 — consequential external effect
A5 — privileged / high-blast-radius effect

A task may be inside the envelope for A1 but outside it for A4.

For example:

Task:
    propose a database migration

Evidence:
    strong benchmark coverage
    schema snapshot available
    migration linting available

Authority:
    A1 — propose

The same agent may not be trusted to execute that migration automatically:

Task:
    execute database migration in production

Evidence:
    no reliable rollback verifier
    large blast radius
    historical production coverage sparse

Authority:
    human-before-commit

Competence and authority should therefore be coupled but not conflated.

Start With Task Classification

Before checking whether a task is inside the competence envelope, the platform needs to classify the task.

Do not let the model produce one vague label such as:

"coding"

That is too coarse.

A useful task descriptor might include:

from dataclasses import dataclass
from typing import Literal


@dataclass(frozen=True)
class TaskDescriptor:
    domain: str
    task_family: str
    operation_kind: str
    environment: str
    risk_class: Literal["low", "medium", "high", "critical"]
    reversibility: Literal["easy", "bounded", "difficult", "irreversible"]
    external_effects: bool
    privileged_access: bool
    estimated_scope: int | None
    required_tools: tuple[str, ...]
    required_verifiers: tuple[str, ...]

For a coding task:

domain: software_engineering
task_family: dependency_upgrade
operation_kind: repository_change
environment: isolated_worktree
risk_class: medium
reversibility: easy
external_effects: false
privileged_access: false
required_tools:
    - repository
    - package_manager
required_verifiers:
    - unit_tests
    - dependency_audit

For a deployment task:

domain: devops
task_family: production_deployment
operation_kind: production_mutation
environment: prod-eu-west
risk_class: high
reversibility: bounded
external_effects: true
privileged_access: true
required_tools:
    - deployment_api
    - observability
required_verifiers:
    - health_checks
    - rollout_metrics
    - rollback_validation

The envelope evaluator now has structured dimensions to compare.

Build the Envelope From Evidence, Not Intent

A dangerous pattern is:

we designed the agent for this
therefore it is competent at this

Design intent is not evidence.

The competence envelope should be built from observed performance.

Useful evidence includes:

held-out benchmark results
production verified outcomes
shadow evaluations
canary results
incident history
failure-injection tests
counterfactual replay
human-review rescue rates
verifier coverage
false-success measurements

Suppose an agent has:

2,000 verified Python bug-fix runs
500 TypeScript bug-fix runs
40 migration runs
3 production rollback cases
0 cryptographic implementation cases

That evidence does not support one universal statement:

agent reliability = 94%

It supports several local claims with different strengths.

This is why cohorting from Steps 23 and 27 matters.

Competence Is Local

Imagine these benchmark results:

A single global average destroys the useful information.

The correct conclusion may be:

Python bug fix:
    autonomous reversible execution permitted

TypeScript bug fix:
    autonomous reversible execution permitted with stronger verification

DB migration proposal:
    proposal permitted

DB migration execution:
    human-before-commit

Kubernetes incident:
    diagnostic support only

Key rotation:
    outside competence envelope

That is much closer to how professional systems assign authority.

Out-of-Distribution Is Multi-Dimensional

OOD detection is often discussed as though there is one training distribution.

Agent systems have many distributions.

A task can be OOD along one dimension while familiar along another.

Consider:

familiar programming language
unfamiliar repository scale
familiar operation
unfamiliar security context
familiar tool
unfamiliar tool version
familiar task family
unfamiliar blast radius
familiar environment
unfamiliar dependency state

A practical OOD vector might look like:

@dataclass(frozen=True)
class OODSignals:
    task_family_novelty: float | None
    domain_novelty: float | None
    environment_novelty: float | None
    tool_novelty: float | None
    scale_novelty: float | None
    risk_novelty: float | None
    verifier_gap: bool
    unsupported_capability: bool
    historical_coverage: int

But be careful.

These numbers should not become fake probabilities.

They are evidence signals.

If you have not calibrated a value as a probability, do not call it one.

Similarity Is Useful but Dangerous

Embedding similarity can help detect novelty.

For example:

new task
   ↓
embedding
   ↓
nearest historical task clusters
   ↓
distance / density / coverage

But similarity alone is insufficient.

These two tasks may be semantically close:

restart the staging service
restart the production payment service

Yet the authority requirements are completely different.

Likewise:

update dependency in test project
update dependency in medical-device firmware

The code change may look nearly identical.

The operational risk is not.

Therefore:

OOD detection must include operational context, not only semantic similarity.

Use Multiple OOD Signals

A robust competence evaluator should combine several types of evidence.

1. Categorical coverage

Has this exact task family been benchmarked?

known
adjacent
unknown

2. Historical density

How many comparable verified cases exist?

2 cases
≠
2,000 cases

3. Distance from known cohorts

How far is the current task from validated examples?

4. Environment mismatch

Is the current execution environment materially different?

5. Tool mismatch

Are new tools, tool versions, or permission scopes involved?

6. Risk mismatch

Does the task have greater consequence than the historical cohort?

7. Verification mismatch

Is the verifier weaker than the verifier used to establish competence?

8. Distribution drift

Has the workload changed since the competence evidence was gathered?

9. Incident exclusions

Does the task match a known unresolved failure signature?

No one signal needs to be magical.

The value comes from preserving the dimensions.

Verification Strength Is Part of Competence

An agent may appear reliable only because its historical workload had strong verifiers.

For example:

coding cohort
    unit tests
    integration tests
    static analysis
    type checks

may support high autonomous authority.

The same model used for:

open-ended architecture recommendation

may have no comparably strong external verifier.

You cannot transfer the same reliability claim.

So competence evidence should record verifier conditions:

@dataclass(frozen=True)
class CompetenceEvidence:
    cohort_id: str
    release_id: str
    verifier_profile_id: str
    verified_success_rate: float | None
    false_success_rate: float | None
    unknown_rate: float | None
    sample_size: int
    observed_period: str

The runtime can then ask:

Is the current verifier profile at least as strong
as the verifier profile under which competence was demonstrated?

If not, authority may need to contract.

False Success Matters More Than Raw Success

Consider two cohorts.

Cohort A

verified success: 91%
false success: 0.2%
UNKNOWN: 5%

Cohort B

verified success: 95%
false success: 6%
UNKNOWN: 0.5%

If the task is consequential, Cohort A may be far more trustworthy.

This is why competence envelopes must inherit the reliability vector from Step 27.

Do not define competence as:

success_rate > threshold

Use something closer to:

verified success acceptable
AND false-success ceiling satisfied
AND verifier coverage sufficient
AND side-effect integrity satisfied
AND evidence volume sufficient

Define Competence Levels

A practical runtime can use discrete states.

For example:

VALIDATED
SUPPORTED
LIMITED
UNVALIDATED
EXCLUDED

Where:

VALIDATED

Strong direct evidence for the task/risk/verifier regime.

SUPPORTED

Good adjacent evidence, but not enough for maximum authority.

LIMITED

Some useful evidence exists, but human approval or reduced authority is required.

UNVALIDATED

No meaningful evidence that the runtime is reliable for this task class.

EXCLUDED

Known failure mode, prohibited domain, unresolved incident class, or explicit policy boundary.

Notice that:

UNVALIDATED

is not the same as:

FAIL

It means:

we do not have enough evidence to trust autonomous execution

That is a much more honest systems statement.

A Competence Decision Object

Make the result explicit.

from dataclasses import dataclass
from typing import Literal


CompetenceLevel = Literal[
    "VALIDATED",
    "SUPPORTED",
    "LIMITED",
    "UNVALIDATED",
    "EXCLUDED",
]


@dataclass(frozen=True)
class CompetenceDecision:
    level: CompetenceLevel
    allowed_authority: str
    matched_cohorts: tuple[str, ...]
    missing_evidence: tuple[str, ...]
    ood_reasons: tuple[str, ...]
    required_verifiers: tuple[str, ...]
    required_escalation: str | None
    policy_version: str

Now the decision can be logged, replayed, audited, and tested.

It is not hidden inside prompt prose.

Do Not Collapse Everything Into a “Competence Score”

It is tempting to write:

competence = 0.82

Then:

if competence > 0.7:
    proceed

That is easy.

It is also dangerous.

Suppose:

task familiarity = high
historical coverage = high
verifier strength = high
risk mismatch = critical

A weighted average might still be high.

But one hard risk mismatch should potentially block autonomous execution.

Use hard constraints where hard constraints exist.

For example:

if task.risk_class == "critical" and not evidence.critical_risk_validated:
    return CompetenceDecision(
        level="UNVALIDATED",
        allowed_authority="A1",
        matched_cohorts=(),
        missing_evidence=("critical-risk validation",),
        ood_reasons=("risk class exceeds validated envelope",),
        required_verifiers=("independent verifier",),
        required_escalation="dual_control",
        policy_version="competence-v3",
    )

The simplicity is a feature.

Hard Boundaries Before Learned OOD Models

Before building a neural OOD detector, implement obvious deterministic checks.

Examples:

unknown tool family
unsupported file type
repository exceeds validated scale
privileged operation outside validated environments
missing mandatory verifier
unseen external system
critical task with no critical benchmark cohort
known unresolved incident signature

These are cheap, explainable, and auditable.

Only add learned novelty models where deterministic structure is insufficient.

This follows the rule we have used throughout the series:

Add complexity only when you can identify the failure the simpler mechanism cannot handle.

Learned OOD Detectors Have Their Own Competence Problem

Suppose you train a classifier:

task features → in-distribution / out-of-distribution

Now you have another model whose failure can change authority.

That model needs evaluation too.

Important metrics include:

OOD recall
false OOD rate
missed high-risk OOD rate
cohort-specific OOD performance
calibration drift

The most dangerous failure is often:

OOD task
    ↓
classified as familiar
    ↓
authority retained

Therefore a learned OOD detector should usually operate inside deterministic authority limits, not replace them.

Distinguish Novelty From Difficulty

A familiar task can be difficult.

An unfamiliar task can be easy.

These are different axes.

               difficulty
             low       high
novelty low   routine   hard-known
        high  new-easy  new-hard

The runtime response should differ.

Hard but known

You may have evidence that search, stronger models, or escalation works well.

Easy but novel

The task may require only proposal mode until evidence accumulates.

Hard and novel

This is where autonomy should contract aggressively.

This distinction prevents the scheduler from treating every OOD event as simply “spend more compute.”

More compute does not manufacture competence evidence.

Out-of-Distribution Is Not Fixed Forever

The envelope should evolve.

Suppose the agent encounters a new task family:

Rust dependency upgrade

Initially:

UNVALIDATED

The system may permit:

A0 observe
A1 propose
human review
sandbox execution
strong verification

After enough evidence accumulates:

100 sandbox cases
50 shadow cases
25 canary cases
0 false successes
strong deterministic verification

The task family might move to:

SUPPORTED

Then later:

VALIDATED

This is evidence-gated authority expansion.

Not confidence-gated expansion.

Competence Expansion Should Be a Release

Do not let the runtime quietly expand its own envelope because it saw a few successful cases.

Treat envelope changes like behavioral releases.

For example:

candidate envelope change
      ↓
offline evaluation
      ↓
shadow
      ↓
limited authority
      ↓
canary
      ↓
promotion

The envelope itself should be versioned:

competence_policy: competence-v7
release: agent-release-2026-08-09-4
cohort: python-db-migration
previous_level: LIMITED
proposed_level: SUPPORTED
max_authority: A2
required_verifier_profile: db-migration-v3
minimum_cases: 200
false_success_ceiling: 0.005

Step 24’s release engineering applies here directly.

Competence Can Contract Too

The envelope is not monotonic.

Suppose a new incident cluster appears:

large monorepos
Python + generated protobuf
partial integration-test coverage

The system may have previously allowed autonomous A2 execution.

After a false-success spike:

VALIDATED → LIMITED

The corresponding authority policy might become:

A2 → A1 + human approval

This should happen quickly when risk is high.

Remember the asymmetry from Step 29:

Authority should contract faster than it expands.

Connect Competence to Error Budgets

Step 27 introduced SLOs and error budgets.

Competence envelopes should consume that information.

Suppose:

python_bugfix cohort:
    false-success budget healthy

browser_checkout cohort:
    false-success burn 4x

The competence evaluator should not continue treating both cohorts equally.

Possible policy:

HEALTHY
    retain current envelope

WATCH
    stronger verifier required

CONSTRAINED
    reduce maximum authority

FREEZE
    no autonomous external effects

This makes reliability state operational.

Connect Competence to Drift Detection

Step 23 covered behavioral drift.

Suppose a task remains inside the nominal envelope, but the environment has drifted.

Examples:

new model version
new tool API
new browser layout
new repository framework
new package-manager behavior
new deployment topology

The task may now be effectively OOD.

So competence should be bound to versions and conditions:

competence evidence
    model v17
    router v8
    verifier v5
    tool schema v12
    environment family E3

Changing those assumptions can invalidate the envelope.

Competence is not a timeless property of the task label.

Connect Competence to Retrieval

Research agents need a special form of competence checking.

Suppose the historical benchmark used:

well-indexed public sources
high source redundancy
recent documents
clear provenance

Now the runtime receives:

three internal PDFs
partial scans
conflicting dates
missing provenance

The topic may be familiar.

The evidence environment is not.

That should produce something like:

evidence_environment_novelty = high
verification_gap = true
competence = LIMITED

The correct response may be:

summarize evidence
identify conflicts
request missing sources
avoid strong factual conclusions

rather than simply ask a larger model.

Connect Competence to Browser Agents

Browser automation illustrates why semantic familiarity is insufficient.

A browser agent may have excellent evidence for:

search
navigation
read-only extraction
form drafting

and very little evidence for:

financial checkout
account deletion
contract acceptance
privilege changes

All of these use the same browser.

The tool is familiar.

The authority regime is not.

A browser competence envelope should include:

page/action family
external effect type
reversibility
value at risk
identity/account context
available postcondition verifier

Connect Competence to Coding Agents

For coding agents, useful competence dimensions include:

language
framework
repository size
build system
test coverage
change class
security sensitivity
runtime environment
migration involvement
external API impact

Example:

Task A:
    fix null check in Python API

Task B:
    change distributed locking semantics

Both are “code changes.”

They should not share an authority envelope.

A useful coding policy might say:

routine localized change
    strong tests
    reversible workspace
    validated language/framework
        → autonomous candidate + verification

cross-service concurrency semantics
    sparse direct benchmark evidence
    high blast radius
        → proposal + isolated experiment + human review

Connect Competence to DevOps Agents

DevOps competence must be state-sensitive.

The same operation may be familiar under normal conditions and OOD during an incident.

For example:

restart service

During normal operation:

healthy dependency graph
normal traffic
single-region scope
known rollback

During an outage:

multiple dependencies degraded
partial telemetry
regional failover active
queues saturated

The command is the same.

The operating regime is completely different.

So the competence envelope needs environment-state assumptions.

Connect Competence to Multi-Agent Systems

Mixture-of-agents systems add another problem.

A router may send an unfamiliar task to the “closest” specialist.

That does not mean any expert is competent.

A good router should be able to return:

NO_SUPPORTED_EXPERT

instead of forcing a choice.

For example:

route = router.select(task)

if route.coverage == "NONE":
    return escalate(
        reason="no expert has validated competence for task regime"
    )

This is better than:

pick whichever expert has the highest score

when every score is poor.

Search Does Not Eliminate the Envelope

Tree search, MCTS, beam search, evolution, self-consistency, and debate can all improve candidate exploration.

None of them automatically expands competence.

If the task is OOD, searching more branches gives you:

more outputs in an unvalidated regime

That may help.

It may also produce a persuasive failure more efficiently.

Therefore:

OOD
    ≠
allocate more search by default

Use Step 18’s value-of-information logic.

Ask:

What information could move this task back inside a supported envelope?

Maybe the answer is:

retrieve authoritative schema
run sandbox experiment
inspect exact state
obtain stronger verifier
ask domain expert

That is better than blind compute escalation.

A Runtime Decision Pipeline

A practical control path might look like:

incoming task
      ↓
classify task + risk + environment
      ↓
resolve current release / verifier / tool state
      ↓
match historical competence cohorts
      ↓
run hard exclusions
      ↓
compute OOD evidence vector
      ↓
check SLO / drift / incident state
      ↓
derive competence level
      ↓
derive maximum authority
      ↓
run / downgrade / escalate / defer / prohibit

Notice that the LLM is not the final authority.

The policy engine is.

Example Policy

A simple rule system may be enough.

def decide_competence(task, evidence, runtime):
    if task.operation_kind in runtime.prohibited_operations:
        return "EXCLUDED", "A0"

    if evidence.sample_size == 0:
        return "UNVALIDATED", "A1"

    if evidence.false_success_rate is None:
        return "LIMITED", "A1"

    if task.risk_class == "critical" and not evidence.critical_validation:
        return "LIMITED", "A1"

    if runtime.verifier_strength < evidence.required_verifier_strength:
        return "LIMITED", "A1"

    if runtime.error_budget_state in {"CONSTRAINED", "FREEZE"}:
        return "LIMITED", "A1"

    if evidence.direct_match and evidence.sample_size >= 500:
        return "VALIDATED", "A2"

    return "SUPPORTED", "A1"

This is intentionally boring.

That is good.

You can always add sophistication after you prove a failure in the simple policy.

UNKNOWN Must Remain a Valid Result

Sometimes the system cannot tell whether the task is in distribution.

Maybe:

historical metadata is incomplete
cohort labels changed
verifier evidence is missing
current environment cannot be fingerprinted
retrieval provenance is unavailable

Do not silently classify the task as familiar.

Return:

COMPETENCE_UNKNOWN

and contract authority.

The absence of evidence is not evidence of safety.

Competence Evidence Needs Provenance

Step 25’s provenance machinery applies here too.

A competence claim should be traceable to:

benchmark cases
production cohorts
verifier versions
release versions
incident exclusions
measurement window
policy version

For example:

cohort: python-api-bugfix
level: VALIDATED
release: r42
verifier_profile: vp9
sample_size: 1832
window: 2026-06-01..2026-08-01
false_success_rate: 0.003
unknown_rate: 0.021
max_authority: A2
excluded_signatures:
  - distributed-locking-change
  - auth-token-rotation

Now the statement “this agent is validated for Python API bug fixes” has an evidence trail.

Competence Must Be Recomputed After Significant Releases

Step 24 made agent behavior a release bundle.

A major change to:

model
prompt
router
search policy
verifier
retrieval
memory
runtime

can invalidate old competence evidence.

Do not assume competence transfers automatically.

You may allow evidence carry-over when compatibility has been demonstrated.

But that itself should be explicit.

For example:

old release cohort evidence
      ↓
compatibility replay
      ↓
paired benchmark comparison
      ↓
carry-over approved

Otherwise the new release starts with reduced authority.

Use Negative Evidence

Competence envelopes should not be built only from successes.

Known failures are extremely valuable.

Suppose incident forensics identified this failure signature:

repository > 1M LOC
partial generated code
multiple package managers
stale dependency graph

That should become an explicit exclusion or authority contraction rule.

if signature matches:
    VALIDATED → LIMITED

This is much stronger than hoping the model “learned from the incident.”

Measure Envelope Quality

You need to evaluate the competence system itself.

Useful metrics include:

Coverage

% of incoming tasks inside VALIDATED/SUPPORTED envelope

OOD detection recall

% of later-confirmed OOD incidents caught before autonomous execution

False OOD rate

% of safe familiar tasks unnecessarily downgraded

False in-distribution rate

% of unvalidated tasks incorrectly granted familiar authority

This is especially important.

Escalation rescue rate

% of competence-driven escalations where human/stronger process prevented a failure

Unnecessary escalation rate

% of escalations where autonomous execution would have been verified safe

Authority-regret rate

cases where granted authority later appears too high

Envelope expansion quality

newly promoted cohort performance vs expected performance

Envelope contraction latency

time from drift/incident evidence to authority reduction

Test With Deliberate OOD Cases

Do not wait for production novelty.

Build OOD test suites.

For a coding agent:

new language
new build system
very large repository
missing tests
security-sensitive code
concurrency-sensitive code
corrupted repository state
new tool version

For browser agents:

unexpected authentication flow
financial transaction page
new consent workflow
captcha / bot defense
ambiguous submit button
irreversible account operation

For research agents:

single-source evidence
contradictory sources
stale corpus
untrusted documents
missing provenance
novel domain terminology

For DevOps agents:

multi-region outage
partial telemetry
control-plane degradation
stale configuration snapshot
unexpected dependency graph
privileged emergency operation

The test is not simply:

can the agent solve it?

The more important test may be:

did the runtime recognize that autonomous authority was not justified?

Failure Mode: Confidence Masquerading as Competence

The agent says:

I am 95% confident.

The runtime proceeds.

This is unacceptable unless that confidence is actually calibrated and relevant to the task regime.

A better rule:

self-confidence can influence search
but cannot expand hard authority limits

Confidence is an internal signal.

Competence is an evidence claim.

Failure Mode: Nearest Neighbor = Supported

The task looks similar to previous tasks in embedding space.

The runtime declares it in-distribution.

But the new task has:

higher risk
new tools
weaker verification
larger blast radius

Similarity should produce:

candidate supporting evidence

not:

authority grant

Failure Mode: The Benchmark Island

A system performs beautifully on a fixed benchmark.

Production gradually changes.

The benchmark remains static.

The agent continues claiming competence based on evidence from a disappearing workload.

This is why competence evidence needs:

production cohorts
rolling validation
drift detection
version binding

Benchmarks are evidence.

They are not eternal certification.

Failure Mode: OOD Detector Becomes the New Oracle

You build a sophisticated novelty model.

Soon the runtime trusts:

ood_score = 0.17

without understanding where it came from.

Now the safety boundary depends on another opaque model.

Keep deterministic exclusions and hard risk rules outside it.

Use the detector as evidence, not sovereign authority.

Failure Mode: Competence Only Expands

Successful cases accumulate.

The envelope grows.

But it never shrinks after:

model drift
verifier drift
tool changes
incidents
new workload regimes

That creates stale authority.

Envelope contraction must be a normal operation.

Failure Mode: Every OOD Task Goes to a Human

That sounds safe.

It can also destroy the system.

Human review capacity is finite.

If every unfamiliar low-risk task becomes an escalation, reviewers become overloaded and rubber-stamping increases.

Use the authority hierarchy.

For example:

OOD + read-only
    → allow observation

OOD + reversible sandbox
    → allow isolated experiment

OOD + consequential write
    → require human approval

OOD + prohibited domain
    → refuse

Competence should contract authority proportionally to risk.

Failure Mode: The Human Approval Expands Competence

A human approves one unfamiliar operation.

The system records:

success

Then treats the whole task class as validated.

That is not enough.

One approved success is evidence.

It is not broad competence.

Keep sample size, cohort scope, verifier strength, and operating conditions explicit.

Failure Mode: Weak Verification Creates Fake Competence

Suppose an agent is evaluated on tasks where success is judged by another similar model.

The benchmark reports 97% success.

Then deterministic checks reveal substantial false success.

The original competence claim was built on a weak measurement system.

Competence evidence is only as strong as its verifier.

This is why Step 23’s verifier-drift work and Step 27’s verifier coverage must feed directly into the envelope.

Failure Mode: Competence Is Treated as a Model Property

Teams say:

Model X is competent at coding.

But production competence depends on the full system:

model
+ prompt
+ tools
+ retrieval
+ state quality
+ router
+ search policy
+ verifier
+ authority controls
+ operating environment

A model swap can change competence.

A verifier outage can change competence.

A tool upgrade can change competence.

A workload shift can change competence.

The competence envelope belongs to the behavioral system, not the foundation model alone.

A Minimal Implementation

You do not need a research project to begin.

Start with four things.

1. Cohort labels

Record task family, risk class, environment, and verifier profile.

2. Reliability evidence

For each cohort, track:

verified success
false success
UNKNOWN
sample size

3. Hard exclusions

Encode known unsupported/high-risk regimes.

4. Authority mapping

Map competence level to maximum authority.

For example:

AUTHORITY_BY_COMPETENCE = {
    "VALIDATED": "A2",
    "SUPPORTED": "A2",
    "LIMITED": "A1",
    "UNVALIDATED": "A1",
    "EXCLUDED": "A0",
}

Then refine from evidence.

A More Complete Architecture

                    task
                     |
              task classifier
                     |
             context fingerprint
                     |
        +------------+-------------+
        |                          |
 competence evidence store     hard policy
        |                          |
 historical cohorts          exclusions
 incidents                   risk limits
 SLO state                   verifier rules
 release bindings            authority caps
        |                          |
        +------------+-------------+
                     |
              OOD evaluator
                     |
          competence decision
                     |
          authority/escalation
             /       |       \
        autonomous  human   refuse
             |        |        |
          execute   review   record
             \        |       /
              verified outcome
                     |
             evidence update

The outcome feeds the evidence store.

But promotion remains governed by release policy.

The runtime does not automatically grant itself more power.

Competence and Expected Value of Information

Step 18 asked:

what should the agent observe next?

Competence checking adds a useful target:

what evidence would move this task from UNVALIDATED to SUPPORTED?

Maybe the missing evidence is:

exact database schema
sandbox execution
stronger tests
current infrastructure topology
independent source
human domain judgment

Now information gathering is tied to authority.

That can make the system dramatically more efficient.

Instead of endlessly “thinking harder,” the agent gathers the specific evidence required to cross a known boundary.

Competence and Dynamic Budgets

Step 16 introduced dynamic compute budgets.

An OOD task should not automatically get unlimited compute.

Instead:

novelty high
    ↓
identify missing competence evidence
    ↓
spend budget on evidence acquisition
    ↓
re-evaluate competence
    ↓
if still unsupported:
    escalate / defer / refuse

This is a better use of compute than generating ten more answers to the same unfamiliar problem.

Competence and Incident Forensics

Step 26 can now ask a new question after incidents:

Was the task outside the competence envelope before execution?

Possible answers:

YES — OOD detected but authority policy failed
YES — OOD detector missed it
NO — task was in validated envelope, execution failed
UNKNOWN — historical competence evidence incomplete

That distinction matters for remediation.

If the task was obviously outside the envelope, improving the model may be the wrong fix.

The fix may be:

better detection
stronger authority contraction
better escalation
better policy enforcement

Competence and Reliability Economics

Step 28 prioritized reliability investments.

Competence envelopes add another measurable remediation surface.

Suppose many false-success incidents share:

low historical cohort density
high environment novelty
weak verifier match

You may discover that improving competence detection provides more reliability return than upgrading the model.

That is exactly the kind of comparison the series is trying to make possible.

Competence Is a Claim, Not a Feeling

This is perhaps the most important idea in the post.

When a system says:

this task is inside the competence envelope

that statement should mean:

we have versioned evidence
on sufficiently comparable tasks
under sufficiently comparable conditions
with sufficiently strong verification
showing that this authority level meets our reliability contract

That is a defensible claim.

It can be audited.

It can be disproved.

It can be updated.

It can shrink.

It can grow.

It is far stronger than:

the model seems confident

What to Build First

If your production agent currently has no competence-envelope machinery, do not begin with sophisticated density estimation.

Start here:

1. classify tasks into meaningful cohorts
2. attach verified outcomes to those cohorts
3. record verifier strength
4. define hard unsupported regimes
5. map cohort evidence to maximum authority
6. return UNVALIDATED when evidence is absent
7. require human approval for consequential OOD actions
8. track false in-distribution and false OOD decisions

That already gives you a large improvement over confidence-based autonomy.

Then add learned novelty detection only where the simpler system demonstrably misses important cases.

The Architecture So Far

The advanced-agent stack now looks less like a clever prompt loop and more like a production control system:

models / tools / memory / search
            ↓
trajectory observability
            ↓
controlled adaptation
            ↓
policy optimization
            ↓
dynamic budgeting
            ↓
typed uncertainty
            ↓
value of information
            ↓
speculative execution
            ↓
distributed coordination
            ↓
platform scheduling
            ↓
failure containment
            ↓
behavioral drift detection
            ↓
release engineering
            ↓
replay + provenance
            ↓
incident forensics
            ↓
SLOs + error budgets
            ↓
reliability prioritization
            ↓
authority boundaries
            ↓
competence envelopes

Notice what happened.

The model moved further and further away from being the sole authority.

That is not accidental.

A production agent becomes more trustworthy not because the model becomes omniscient, but because the system becomes better at knowing:

what it knows
what it can verify
where it has evidence
where it does not
what it may do
when it must stop

The Next Problem: Safe Exploration Outside the Envelope

Competence envelopes create an immediate next question.

If a task is outside the validated envelope, do we simply stop forever?

No.

A useful system needs a disciplined way to explore new territory without granting production authority prematurely.

That means building sandboxed capability acquisition:

OOD task
   ↓
restricted sandbox
   ↓
experiments / probes / simulation
   ↓
strong verification
   ↓
new evidence
   ↓
offline competence update proposal
   ↓
shadow / canary
   ↓
possible envelope expansion

The key idea will be:

Learn outside the authority boundary before expanding the authority boundary.

That is where we go next.