The agent says:

Done.

That is a claim.

It is not evidence.

Every mechanism in this book so far has made the agent better at deciding what to do, and none of them establishes that the user’s goal was achieved. A planner can produce a coherent plan for the wrong problem. A tool can return exit code zero without producing the intended effect. A search can select the highest-scoring branch when every branch is wrong. A memory system can retrieve a perfectly relevant fact that stopped being true in March.

A model can narrate all of this fluently while reality sits unchanged.

The last mechanism is therefore the one that decides what any of it earned:

Verification is an evidence-and-integrity boundary between the agent and the claim that the goal was achieved.

Two words in that sentence are doing unequal amounts of work. Evidence is the familiar half, and most teams get some version of it right. Integrity becomes essential once an agent can act on the environment that grades it, because the question is no longer only whether the check passed. It is whether passing the check still required doing the task.

1. Four things that all look like success

A deployment agent runs kubectl rollout restart deployment/api and gets exit code zero.

What has been established?

That one command was accepted. Not that new pods became ready, that health checks pass, that the error rate returned to baseline, that the requested version is actually the one now serving traffic, or that no protected constraint was broken along the way.

Four distinct levels hide inside the word “success”, and conflating them is the root of most false completions.

These are not interchangeable, and each is cheap to mistake for the one below it.

A successful receipt routinely accompanies a failed task. A genuinely correct final state can sit on top of an invalid evaluation path, in which case the system got the right answer and learned nothing it can rely on next time.

Do not promote lower-level success into higher-level success without evidence for the missing layer.

The architecture already points here. The action boundary established that a model’s output is a proposal rather than authority to execute. The same demotion now applies at the other end of the run: the model saying “done” is a completion proposal, and the runtime decides whether the evidence bar was met.

The model proposes. The runtime controls execution. The environment supplies evidence. The verifier decides what that evidence earns.

    flowchart TD
    G[user goal] --> C[goal contract]
    C --> A[agent acts]
    A --> E[environment changes]
    E --> EV[evidence collection]
    EV --> B[state binding + integrity checks]
    B --> AD[protected adjudication]
    AD --> V{verdict}
    V -->|PASS| S[stop]
    V -->|FAIL| R[diagnose and recover]
    V -->|PARTIAL| P[pursue remainder]
    V -->|UNKNOWN| Q[gather evidence or escalate]
    R --> A
    P --> A
    Q --> EV
  

2. The goal contract

A verifier cannot verify an undefined goal, and most goals arrive underdefined in a specific way: they state what should change and leave implicit what must not.

Add pagination without changing the existing response schema.

A weak agent reduces that to “pagination works” and drops the preservation constraint, which was the harder half of the request.

The fix is to represent the goal as a structure before any work starts.

from collections.abc import Callable
from dataclasses import dataclass, replace
from enum import StrEnum


class CriterionKind(StrEnum):
    CHANGE = "change"        # something must become true
    PRESERVE = "preserve"    # something must remain true
    FORBID = "forbid"        # something must not become true


@dataclass(frozen=True)
class Criterion:
    id: str
    description: str
    kind: CriterionKind
    required: bool = True


@dataclass(frozen=True)
class GoalContract:
    criteria: tuple[Criterion, ...]
    revision: int = 0

    @property
    def required_ids(self) -> frozenset[str]:
        return frozenset(c.id for c in self.criteria if c.required)

For the pagination request that becomes a change criterion for pagination, preserve criteria for the response schema and authentication behaviour, and a forbid criterion covering removal or bypass of tests. Success now means satisfying the requested change while preserving the constraints that made the change acceptable.

Contracts sometimes need to move. An agent may discover that a criterion is impossible, irrelevant, or specified against a misunderstanding, and that is a legitimate finding.

What it is not is a licence to quietly delete the criterion and pass.

The dangerous sequence is short: criterion fails, agent concludes the criterion was unnecessary, criterion disappears, agent passes. Every step there is locally reasonable, which is what makes it hard to spot in a trace.

@dataclass(frozen=True)
class ContractRevision:
    criterion_id: str
    old_description: str
    new_description: str | None   # None removes the criterion
    reason: str
    authorized_by: str


def apply_revisions(
    contract: GoalContract,
    revisions: tuple[ContractRevision, ...],
    authorize: Callable[[ContractRevision], bool],
) -> GoalContract:
    by_id = {c.id: c for c in contract.criteria}
    order = [c.id for c in contract.criteria]
    applied = 0

    for revision in revisions:
        current = by_id.get(revision.criterion_id)
        if current is None:
            raise KeyError(f"unknown criterion {revision.criterion_id!r}")

        if current.description != revision.old_description:
            raise ValueError(
                f"stale revision for {revision.criterion_id!r}: "
                "contract changed since the revision was proposed"
            )

        if not authorize(revision):
            raise PermissionError(
                f"unauthorized revision to {revision.criterion_id!r}"
            )

        if revision.new_description is None:
            del by_id[revision.criterion_id]
            order.remove(revision.criterion_id)
        else:
            by_id[revision.criterion_id] = replace(
                current,
                description=revision.new_description,
            )

        applied += 1

    return GoalContract(
        criteria=tuple(by_id[cid] for cid in order),
        revision=contract.revision + applied,
    )

Three details are deliberate. authorized_by is audit data, not proof of authority; the trusted runtime decides whether that actor may revise the contract. The revision is rejected if it was proposed against an older description, which prevents a stale approval from silently editing a newer contract. And changing a description preserves the criterion’s kind and required status rather than recreating it with defaults.

Changing the definition of success is a separate, attributable action from satisfying it.

The verifier evaluates the current authorized contract, not whichever target turned out to be reachable.

3. Four verdicts, and why UNKNOWN is one of them

Real environments are noisy.

Tasks are partially completed, verifiers cannot always observe enough to decide, and sometimes the verifier itself falls over.

class Verdict(StrEnum):
    PASS = "pass"
    FAIL = "fail"
    PARTIAL = "partial"
    UNKNOWN = "unknown"

FAIL and UNKNOWN are the pair that systems most often collapse, and they mean opposite things. One says the requirement is false. The other says we could not establish whether the requirement is true, which happens whenever monitoring data has not arrived, a third-party API timed out, a required source is unavailable, a replica is too stale to adjudicate, or human approval is still pending.

Manufacturing certainty out of missing evidence is the easiest route to false success there is.

PARTIAL earns its place for a different reason, which is that long-horizon tasks routinely end with meaningful incomplete work. Eight of ten requested migrations completed, two blocked by unavailable credentials, is not honestly described by either PASS or FAIL. Calling it FAIL discards the eight. Calling it PASS is untrue.

Reporting it as partial, with the count and the blocking reason, lets the surrounding system pursue exactly what remains.

Partial success does not relax required criteria. It describes precisely what was and was not established.

Absence of evidence is not evidence of success.

4. Evidence is an object, not a boolean

Reducing verification to success = True throws away everything needed to trust it later.

from datetime import datetime


@dataclass(frozen=True)
class Evidence:
    criterion_id: str
    source: str
    tier: "EvidenceTier"
    state_id: str
    collected_at: datetime
    verifier_version: str
    verdict: Verdict
    payload: dict[str, object]


@dataclass(frozen=True)
class CriterionResult:
    criterion_id: str
    verdict: Verdict
    evidence: tuple[Evidence, ...] = ()
    reason: str = ""

With that in place the system can answer what was checked, against which state, by which verifier version, using what, when, and what that check actually established. So when the agent reports that the tests passed, the useful question has an answer: which tests, against which state?

That second half is where stale verification lives, and it is a frequent agent failure.

Tests pass. The agent edits the code again. The agent reports the earlier passing result.

The evidence is entirely real, and it describes a state that no longer exists.

state_id is what closes that gap, and what belongs in it varies by domain.

def evidence_is_current(evidence: Evidence, current_state_id: str) -> bool:
    return evidence.state_id == current_state_id

The identifier has to come from trusted runtime or environment state, not from the acting model, and it has to change whenever criterion-relevant state changes. Otherwise the binding exists only syntactically.

If the state changed after collection, the corresponding result reverts to unverified. This is the same temporal rule that governs memory: a cached PASS does not outrank a workspace that has changed since, and a service that was healthy at 14:02 is not evidence about a health endpoint failing at 14:08.

5. Verify the state, not the narrative

Agent evaluation has been moving steadily toward state-based verification, which means grading what the environment became rather than what the agent said about it. STAGE-Claw scores agents through the correctness of the final system state in realistic personal-computing environments rather than through textual output alone.[1]

The translation to production is direct, and it is usually a question of finding the record that a third party would consult.

Prefer evidence about the resulting environment state over evidence about the agent’s narrative of it.

6. Collect, then adjudicate

Evidence collection and adjudication are different jobs and should be different code, because observation failure and judgement failure need different fixes.

A missing service receipt is not a receipt showing failure. A correct receipt interpreted against the wrong criterion is not an environment problem.

For the deployment, the collector reports the exit code, the running digest, ready replicas at three of three, a health endpoint returning 200, and an error rate at baseline. The adjudicator maps those onto the contract’s criteria. Neither knows much about the other. REDAgentBench makes a related separation in safety evaluation, distinguishing actual harmful effects from what the evaluation view can observe, and grounding verdicts in service receipts and final-state changes.[5]

Adjudication has one rule that matters more than the rest, which is that required evidence gates must not dissolve into an average.

Suppose unit tests pass, the API contract test fails, and a style judge passes.

That is not two out of three. The contract failure may be the entire point of the task, and any scalar that lets two soft passes outvote one hard failure has destroyed the meaning of the result.

def adjudicate(
    contract: GoalContract,
    results: dict[str, CriterionResult],
) -> Verdict:
    required = [
        results[c.id]
        for c in contract.criteria
        if c.required and c.id in results
    ]
    missing = contract.required_ids - results.keys()

    if any(r.verdict is Verdict.FAIL for r in required):
        return Verdict.FAIL
    if missing or any(r.verdict is Verdict.UNKNOWN for r in required):
        return Verdict.UNKNOWN
    if any(r.verdict is Verdict.PARTIAL for r in required):
        return Verdict.PARTIAL
    return Verdict.PASS

Note the ordering. A definite failure on any required criterion beats everything, because a task with a broken hard requirement is not partially done. Missing evidence outranks partial evidence, because not knowing is weaker than knowing incompletely.

An earlier draft of this chapter selected required criteria by testing whether the criterion ID started with "required:", and admitted in the surrounding prose that a real implementation would use metadata instead. It should use Criterion.required, which the contract already carries.

7. The verifier is part of the attack surface

Once an agent can affect the environment that grades it, a comfortable assumption breaks: a passing checker is no longer automatically evidence that the intended capability was exercised.

Suppose the task is to get a benchmark score to 0.90 or above, and the agent can modify score.py. A function body of return 1.0 satisfies the grader completely. The same shape recurs constantly: read hidden answers, inspect benchmark metadata, delete failing tests, special-case the visible instances, or skip a verification stage entirely.

The verifier says PASS. The goal was not achieved.

Nothing in that verdict alone distinguishes the run from a legitimate one.

So passing a verifier is necessary and not sufficient, and a second question has to be asked alongside it:

Was the verifier passed through a path that still required the intended capability?

That question needs its own state, tracked separately from correctness.

class IntegrityStatus(StrEnum):
    CLEAN = "clean"
    VIOLATED = "violated"
    UNKNOWN = "unknown"

A criterion PASS with integrity CLEAN can contribute to success. A criterion PASS with integrity VIOLATED and the reason “test file modified by agent” cannot. Integrity UNKNOWN is different again: it means the runtime has not established that the evidence path remained trustworthy, so a positive claim must be downgraded to uncertainty rather than promoted or invented.

The evidence that this matters in practice is now specific. The 2026 Reward Hacking Benchmark studies tool-using agents exploiting evaluation shortcuts, including skipped verification, task-adjacent metadata and tampering with evaluation-relevant functions.[2] A June 2026 audit of 1,968 tasks across five terminal-agent benchmarks found 323 whose outcome verifiers were hackable by frontier models from the task description alone, and introduced an adversarial hacker-fixer hardening loop.[3] A July 2026 protocol-validity study argues that a benchmark score supports a capability claim only when the evaluation protocol keeps the intended capability necessary for success.[4]

That framing generalises past benchmarks. The production version of the question is not only whether the checker returned PASS, but whether the checking path still measures the property anyone cares about.

When an agent optimises against a measurable signal, the measurement path itself needs protection.

8. The minimum evidence bar

Protection means the acting policy does not get unrestricted authority over the things that make evidence meaningful: which criteria are required, what counts as adequate evidence for each, the verifier implementation, reference answers, protected fixtures, evidence logs, and the integrity policy itself.

    flowchart TD
    subgraph M[agent-mutable]
        W[workspace] --> ENV[environment state]
    end
    ENV -.observed by.-> P
    subgraph P[protected: agent cannot write]
        BAR[evidence bar] --> CH[independent checks]
        CH --> ADJ[adjudication]
        ADJ --> INT[integrity monitor]
    end
    style P fill:#eef3fb,stroke:#4a6fa5
  

None of this requires the verifier to be secret. It requires the runtime to preserve whatever separation makes the evidence mean something, which in different systems means hidden tests, immutable policy rules, read-only fixtures, independent service receipts, or a second environment the agent cannot touch.

The subtler attack does not tamper with anything.

It simply picks weaker checks.

Asked to fix an API behaviour, a model-controlled verification plan will often propose reading the edited function rather than reproducing the API request. The first is cheap and self-confirming. The second is causally relevant.

Nothing has been sabotaged. The evidence has just been quietly downgraded, and the verdict can look identical.

The defence is to make the floor a first-class object that the agent can raise and cannot lower.

from enum import IntEnum


class EvidenceTier(IntEnum):
    JUDGE = 0            # model or subjective evaluator
    PROCESS_MODEL = 1    # learned evaluator
    DOMAIN_SIGNAL = 2    # task-specific heuristic or partial check
    ENVIRONMENT = 3      # direct observation, receipt, or deterministic test


@dataclass(frozen=True)
class EvidenceBar:
    """The floor for one criterion. Set by trusted runtime policy."""
    criterion_id: str
    min_tier: EvidenceTier
    accepted_sources: frozenset[str] = frozenset()

    def qualifying(self, evidence: tuple[Evidence, ...]) -> tuple[Evidence, ...]:
        return tuple(
            e for e in evidence
            if e.tier >= self.min_tier
            and (not self.accepted_sources or e.source in self.accepted_sources)
        )

    def cleared_by(self, evidence: tuple[Evidence, ...]) -> bool:
        return bool(self.qualifying(evidence))

The tier ordering is a default trust policy for this runtime, not a universal theorem about evidence. A domain may need a different ordering or source-specific rules, which is why the bar also names accepted sources. The important property is that the minimum bar is explicit, protected and criterion-specific.

The vocabulary is shared with partial-state evaluation during search, but the semantics here are stricter. Search estimates which branch deserves more compute; verification decides what evidence is strong enough to close a criterion. The agent may add checks above the bar freely.

It cannot substitute convenient evidence for required evidence.

9. The verifier, assembled

Coverage has to be explicit, because a contract with five required criteria of which two were checked and passed is not a pass. It is two established and three unknown, and a verifier that does not track the difference will report the wrong thing with complete confidence.

The assembled verifier also has to resolve evidence deliberately. Lower-tier evidence is useful for diagnosis, but it should not be able to outvote a stronger source merely because more copies of it exist.

from typing import Callable


def verdict_from(supporting: tuple[Evidence, ...]) -> Verdict:
    """Adjudicate the strongest qualifying evidence tier conservatively."""
    if not supporting:
        return Verdict.UNKNOWN

    strongest = max(e.tier for e in supporting)
    strongest_evidence = tuple(e for e in supporting if e.tier == strongest)
    verdicts = {e.verdict for e in strongest_evidence}

    if Verdict.FAIL in verdicts:
        return Verdict.FAIL
    if Verdict.UNKNOWN in verdicts:
        return Verdict.UNKNOWN
    if Verdict.PARTIAL in verdicts:
        return Verdict.PARTIAL
    if verdicts == {Verdict.PASS}:
        return Verdict.PASS
    return Verdict.UNKNOWN


@dataclass(frozen=True)
class GoalVerification:
    verdict: Verdict
    integrity: IntegrityStatus
    results: tuple[CriterionResult, ...]
    coverage: float
    established: int
    total_required: int
    reason: str = ""


def verify_goal(
    contract: GoalContract,
    evidence: tuple[Evidence, ...],
    bars: dict[str, EvidenceBar],
    *,
    current_state_id: str,
    integrity: IntegrityStatus,
) -> GoalVerification:
    missing_bars = contract.required_ids - bars.keys()
    if missing_bars:
        raise ValueError(
            f"required criteria have no protected evidence bar: "
            f"{sorted(missing_bars)}"
        )

    fresh = tuple(
        e for e in evidence
        if evidence_is_current(e, current_state_id)
    )

    results: dict[str, CriterionResult] = {}

    for criterion in contract.criteria:
        supporting = tuple(
            e for e in fresh
            if e.criterion_id == criterion.id
        )
        bar = bars.get(criterion.id)

        if not supporting:
            continue

        qualified = (
            bar.qualifying(supporting)
            if bar is not None
            else supporting
        )

        if not qualified:
            results[criterion.id] = CriterionResult(
                criterion_id=criterion.id,
                verdict=Verdict.UNKNOWN,
                evidence=supporting,
                reason="evidence exists but does not clear the protected bar",
            )
            continue

        results[criterion.id] = CriterionResult(
            criterion_id=criterion.id,
            verdict=verdict_from(qualified),
            evidence=supporting,
        )

    verdict = adjudicate(contract, results)

    if integrity is IntegrityStatus.VIOLATED:
        verdict = Verdict.FAIL
    elif (
        integrity is IntegrityStatus.UNKNOWN
        and verdict in (Verdict.PASS, Verdict.PARTIAL)
    ):
        verdict = Verdict.UNKNOWN

    required = contract.required_ids
    established = sum(
        1 for cid in required
        if cid in results and results[cid].verdict is not Verdict.UNKNOWN
    )

    reason = ""
    if integrity is IntegrityStatus.VIOLATED:
        reason = "verification integrity violated"
    elif integrity is IntegrityStatus.UNKNOWN:
        reason = "verification integrity not established"
    elif verdict is not Verdict.PASS:
        reason = "see criterion results"

    return GoalVerification(
        verdict=verdict,
        integrity=integrity,
        results=tuple(results.values()),
        coverage=established / max(len(required), 1),
        established=established,
        total_required=len(required),
        reason=reason,
    )

The interesting part is not the type hierarchy. It is the list of things this function refuses to collapse into each other: missing evidence, stale evidence, evidence below the bar, disagreement among equally trusted evidence, failed evidence, partial evidence, and an integrity violation.

A missing evidence bar for a required criterion is treated as a verifier configuration error rather than as permission to accept whatever evidence happens to exist.

The integrity check can only make a positive conclusion harder to earn. VIOLATED yields FAIL. UNKNOWN does not masquerade as a clean path; it downgrades PASS or PARTIAL to UNKNOWN. A definite criterion failure remains a failure regardless.

10. Preconditions and postconditions

The action boundary already owns preconditions: before a proposed action executes, the runtime checks whether the world is in a state where that action may run. Verification does not take that responsibility back.

What verification adds is the other side of the operation: evidence about whether execution produced the state it was supposed to produce. The same registered checks can support both boundaries, but ownership remains different.

@dataclass(frozen=True)
class Gate:
    id: str
    description: str
    check: str   # identifier of a registered deterministic check


@dataclass(frozen=True)
class GatedOperation:
    name: str
    preconditions: tuple[Gate, ...]
    postconditions: tuple[Gate, ...]

For a destructive migration, the action boundary checks preconditions such as schema version 41, a valid backup and a held migration lock before execution. Verification checks postconditions such as schema version 42, reconciled row counts, intact constraints and healthy application state afterwards.

The model may propose the migration. The runtime owns the precondition gate before execution and the postcondition evidence path after it. A failed postcondition is a verified failure rather than an unfortunate outcome to be narrated.

The useful unification is in the check registry, not in collapsing the two stages. The same trusted check can participate in precondition enforcement, progress measurement and final verification while each mechanism still answers its own question.

11. Staging by cost and consequence

Verification consumes time, tool calls and sometimes money, so the order in which checks run is itself a policy. A verifier should not spend fifteen minutes proving that code which does not parse also fails the full regression suite.

def verify_staged(
    stages: list[tuple[str, Callable[[], tuple[Evidence, ...]]]],
    contract: GoalContract,
    bars: dict[str, EvidenceBar],
    *,
    current_state_id: str,
    integrity: IntegrityStatus,
) -> GoalVerification:
    collected: tuple[Evidence, ...] = ()

    if not stages:
        return verify_goal(
            contract,
            collected,
            bars,
            current_state_id=current_state_id,
            integrity=integrity,
        )

    result = verify_goal(
        contract,
        collected,
        bars,
        current_state_id=current_state_id,
        integrity=integrity,
    )

    for _name, collect in stages:
        collected += collect()
        result = verify_goal(
            contract,
            collected,
            bars,
            current_state_id=current_state_id,
            integrity=integrity,
        )
        if result.verdict is Verdict.FAIL:
            return result          # a hard failure ends the cascade

    return result

For code the cascade might run parse, then targeted reproduction, then related tests, then the full suite, then performance checks if the contract asks for them. For data it might run schema, sample invariants, partition reconciliation, then full reconciliation. The universal part is not the order but the shape: spend the evidence budget in proportion to failure probability and consequence, and stop as soon as a required criterion is definitively broken.

Only FAIL short-circuits by default.

An UNKNOWN at stage two is a reason to keep collecting, not a reason to stop. PARTIAL can also justify continuing when later stages may establish the remainder. A production verifier may add cost ceilings or consequence-specific escalation, but those should be explicit policies rather than accidental early exits.

12. Judges are evidence, not oracles

Not every task has a deterministic oracle. Writing a persuasive memo, reviewing an architecture, synthesising a literature, improving prose: none of these has a test suite, and pretending otherwise produces theatre.

Verification does not disappear there.

It decomposes, because subjective tasks contain objective islands.

A research synthesis has entirely checkable properties. The cited sources exist and are reachable. The dates are right. The quoted text matches the source. The claims do not exceed what the cited evidence supports. The requested sections are present. What remains genuinely subjective after all of that is clarity, synthesis quality and judgement, which is a much smaller residue than the task looked like at the start.

@dataclass(frozen=True)
class Claim:
    id: str
    text: str
    source_ids: tuple[str, ...]


def verify_claims(
    claims: tuple[Claim, ...],
    check_source,
) -> dict[Verdict, int]:
    from collections import Counter
    return Counter(check_source(claim) for claim in claims)

Reporting four claims supported, one partially supported and one unsupported is far more useful than an answer quality of 8.7 out of 10, because the first can be acted on and the second cannot.

For the residue, a learned judge is a reasonable instrument as long as its output is read correctly. “The judge says PASS” means an evaluator produced evidence supporting PASS. It does not mean reality was proven. Under the default policy above, a judge sits below direct environment evidence and cannot clear a criterion whose protected bar requires an environment source, however confident it sounds.

Judges improve with an explicit rubric, grounding in the source material, separation of the generator and judge roles, adversarial examples, and a calibration set with known labels. They should also be measured, on independently labelled cases, for the same two error rates as any other verifier.

13. Verification drives recovery

A verdict is not only a final report. It is an input to the control loop, and each of the four leads somewhere different: PASS stops, FAIL diagnoses the mismatch and recovers, PARTIAL pursues the remaining criteria, and UNKNOWN gathers more evidence, retries the observation, or escalates.

For any of that to work, a failure has to be structured enough to change the next decision. A bare FAIL is not. A criterion of checkout_total with an expected value of 49.99, an observed value of 59.99, a state of order:8172:v6 and a source of the backend order record is a diagnosis with a next step attached.

There is one loop to be careful about, because it manufactures certainty out of noise:

def verify_with_budget(run_check, *, attempts: int = 3) -> Verdict:
    seen = [run_check() for _ in range(attempts)]

    if any(v is Verdict.FAIL for v in seen):
        return Verdict.FAIL
    if all(v is Verdict.PASS for v in seen):
        return Verdict.PASS
    if all(v is Verdict.PARTIAL for v in seen):
        return Verdict.PARTIAL
    return Verdict.UNKNOWN

The failure mode this prevents is the one where UNKNOWN is retried until a PASS appears and the PASS is then treated as the answer. Aggregation must be conservative: any FAIL sticks, a single PASS among unknowns does not clear the criterion, and repeated PARTIAL results remain partial rather than being promoted.

When a deterministic or authoritative source is unavailable, running a model judgement three more times does not substitute for the missing source.

After the evidence budget is spent, UNKNOWN stays unknown.

The same honesty applies to human review, which belongs in this list rather than outside it. Publication, irreversible high-impact actions, ambiguous acceptance criteria, legal approval and financial authorisation may all have a person as their correct final verifier. AWAITING_REVIEW is therefore a legitimate runtime state rather than an architectural embarrassment.

The goal of agent engineering is not maximum autonomy. It is reliable delegation under explicit authority and evidence.

14. Red-team the verifier

Agents get tested with failure injection.

Verifiers should get the same treatment, for the stronger reason that a verifier will be optimised against.

For a coding verifier, the attacks are concrete and worth running deliberately: delete a failing test, skip test discovery, modify a fixture, hard-code the visible example, return the expected value only for test inputs, edit the scoring script, read a protected answer artifact, or leave stale PASS evidence in a cache. Then ask whether the verifier still rejects the invalid solution.

The 2026 hacker-fixer work makes this systematic. A hacker searches for ways to pass without solving, a fixer patches the verifier, a legitimate solver confirms that real solutions still pass, and the cycle repeats until the attack budget stops finding holes.[3] The second half of that loop is the part usually skipped, and it is what stops verifier hardening from becoming verifier paranoia.

Actively search for ways the verifier can be satisfied without satisfying the goal.

The two decisive error rates are not symmetric in consequence, and they should use the actual ground-truth classes as denominators rather than the verifier’s own predictions.

def verifier_error_rates(
    labelled: list[tuple[Verdict, bool]],
) -> dict[str, float]:
    """Known-outcome cases: (verdict returned, goal truly satisfied)."""
    positive = [(v, truth) for v, truth in labelled if truth]
    negative = [(v, truth) for v, truth in labelled if not truth]

    false_pass = sum(
        1 for verdict, _ in negative
        if verdict is Verdict.PASS
    )
    false_fail = sum(
        1 for verdict, _ in positive
        if verdict is Verdict.FAIL
    )

    return {
        "false_pass_rate": false_pass / max(len(negative), 1),
        "false_fail_rate": false_fail / max(len(positive), 1),
        "unknown_rate": (
            sum(1 for verdict, _ in labelled if verdict is Verdict.UNKNOWN)
            / max(len(labelled), 1)
        ),
        "partial_rate": (
            sum(1 for verdict, _ in labelled if verdict is Verdict.PARTIAL)
            / max(len(labelled), 1)
        ),
    }

False FAIL wastes work.

False PASS terminates the agent and publishes an incorrect success state, which is why a more capable agent paired with a weak verifier can look like it is improving while it is only getting better at satisfying the proxy.

Testing the acting agent and the verifier together makes any change uninterpretable, so use known-outcome cases and separate four capabilities: can the agent solve the task, can the verifier recognise real success, can it reject false success, and can the integrity layer reject gamed success.

The injection suite should cover a successful command with a failed effect, a stale passing test, the wrong environment, a partial write, a correct local result with a regression elsewhere, a missing source, contradictory evidence, a modified test, a modified scorer, answer-key exposure, a verifier crash, and a slow eventually-consistent update.

15. What verification costs

Every mechanism in this book has been asked whether it earns its cost, and verification does not get an exemption.

Holding model, tools, task set, budgets and base prompts fixed, measure true task success against reported success, the two error rates, coverage, integrity violations, cost and latency.

The rung that matters is usually not the one with the highest raw completion rate.

It is the one with the smallest gap between claimed success and verified success.

A full verification cascade may reduce false success while noticeably increasing cost, which is an excellent trade for a production deployment and unnecessary for a disposable draft. The required evidence bar should track the consequences of being wrong, which is a decision about the task rather than about the verifier.

16. Where truth lives

The book now forms one construction, and each mechanism answers a question the previous one exposed.

The progression is not toward a more capable model.

It is toward a more explicit runtime, and the final loop shows how little of the consequential machinery now sits inside the model at all.

    flowchart TD
    G[user goal] --> GC[goal contract]
    GC --> ST[runtime state]
    ST --> MP[model proposes]
    MP --> VA{validation + authorization}
    VA -->|rejected| MP
    VA -->|accepted| EX[execute]
    EX --> OB[observation]
    OB --> PR[progress: continue, recover, stop]
    PR --> ST
    PR --> EC[evidence collection]
    EC --> SB[state binding + evidence bar]
    SB --> IN[integrity monitor]
    IN --> AD[protected adjudication]
    AD --> VD[PASS / FAIL / PARTIAL / UNKNOWN]
  

At the start of this book the model looked like the centre of the system. It is still important, and its role is now legible: it generates proposals and makes decisions inside boundaries it does not own. The runtime owns state, authority, action boundaries, budgets, and the memory, search and verification policies. The environment owns the observations from which evidence is built.

So truth does not live in the model’s confidence, in the elegance of the plan, or in a branch score. It does not even live automatically in one passing test.

Do not ask the agent whether reality changed. Build an evidence path that can show what changed, what remained true, and whether the path to that conclusion was itself trustworthy.

When that path produces enough, the verdict is PASS. When it shows the requirement is false, FAIL. When only part of the contract was earned, PARTIAL. And when the system genuinely cannot establish which of those is true, UNKNOWN is a better engineering result than invented certainty.

Research roots

These references establish the ancestry for the mechanisms used here. The chapter’s architecture is an engineering synthesis rather than a claim that any single paper defines the correct agent verifier.

1. Liang et al. — STAGE-Claw: Automated State-based Agent Benchmarking for Realistic Scenarios (2026). Evaluates agents by the correctness of resulting system state in realistic personal-computing environments rather than only textual responses; cited for the state-based verification argument in section 5. https://arxiv.org/abs/2606.10394
2. Thaman — Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use (2026). Studies tool-using agents exploiting evaluation shortcuts including skipped verification, task-adjacent metadata and tampering with evaluation-relevant functions; cited for the attack taxonomy in section 7. https://arxiv.org/abs/2605.02964
3. Zhong et al. — Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops (2026). Audits 1,968 tasks across five terminal-agent benchmarks, finds 323 hackable from the task description alone, and introduces the hacker-fixer-solver loop used in section 14. https://arxiv.org/abs/2606.08960
4. Shao et al. — Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI (2026). Argues that benchmark scores support capability claims only when the evaluation protocol keeps the intended capability necessary for success; cited for the production generalisation in section 7. https://arxiv.org/abs/2607.22368
5. Chen et al. — REDAgentBench: Executable Red Teaming and Faithful Measurement of LLM Agent Systems (2026). Separates harmful effects from evidence visibility and verifies effects using service receipts and final-state changes; cited for the collect/adjudicate distinction in section 6. https://arxiv.org/abs/2608.10669

Next: Building the Complete Agent

Ten mechanisms have been built, and each was argued and measured on its own.

That is not the same as showing they work together. Every chapter established its mechanism against a problem chosen to isolate it, which is the right way to explain something and a poor way to find out whether it survives contact with the other nine. Composition failures are their own category: a memory policy that fights the search budget, a verifier whose state binding does not match the checkpoint format search uses for branch isolation, a plan representation that quietly becomes the state it was supposed to describe.

The final chapter takes one small, genuinely broken repository and runs the whole thing end to end. The interesting result there was not the successful repair. It was what happened when the verifier was attacked.