# Building Systems That Distrust Their Models
# Book: Hallucination From First Principles
# Chapter URL: https://programmer.ie/books/hallucination-from-first-principles/15-chapter/
# Prose fingerprint (sha256, normalized): cf24fb9b505143359b78abcfa0151aa61ecb1999e7723df54add607377f0f845
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

The first chapter began with a simple observation:
A language model can produce a fluent answer without possessing a mechanism that proves the answer is true. Fourteen chapters later, that fact has not changed.
The model can still:
invent misbind misattribute ignore decisive context answer without enough evidence accept bad retrieval repair one error by creating another repeat its own stored mistake The final architecture does not make those possibilities disappear.
It changes who has authority over what happens next.
The model proposes. The system decides what survives.
That is the final systems principle of this book.
Environment and canonical reference implementation The end-to-end example in this chapter is backed by a small deterministic controller:
Environment: Python 3.13.5 Dependencies: Python standard library only Canonical implementation: experiments/ch15/distrustful_controller.py Run it with:
python experiments/ch15/distrustful_controller.py The script is not a production framework.
Its purpose is narrower: demonstrate the boundary invariants in executable form.
A complete run through the system Consider one task:
Prepare a Q3 revenue update for the board. The initial evidence snapshot contains:
Q1 revenue = $41.2M Q2 revenue = $43.8M A model proposes:
Q3 revenue was approximately $46 million. The important point is not that $46M is known to be false.
The current evidence simply does not establish it.
The first measurement record is therefore:
CANDIDATE cand_v1 MEASUREMENT support_relation = NOT_ESTABLISHED provenance = UNVERIFIED epistemic_adequacy = INSUFFICIENT_EVIDENCE The assertion policy returns:
ASSERTION POLICY commitment = HOLD next_action = RETRIEVE matched_rule = EVIDENCE_REQUIRED The candidate separately reaches the persistence boundary:
MEMORY ADMISSION admission = QUARANTINE lifecycle = QUARANTINED allowed_uses = DEBUG, REGRESSION_TEST Nothing in the response decision automatically admits the claim to factual memory.
Now suppose the model proposes a side effect:
SEND_BOARD_UPDATE The action policy independently returns:
ACTION POLICY authorization = HOLD requirement = HUMAN_APPROVAL_REQUIRED The enforcement point does not care how persuasive the model sounds.
The reference controller actually raises:
AuthorizationError: action SEND_BOARD_UPDATE requires HUMAN_APPROVAL_REQUIRED; execution blocked No email is sent.
Now retrieval obtains an authoritative Q3 filing:
Q3 revenue = $47.3M The system creates a new evidence snapshot and a new candidate:
cand_v2 Q3 revenue was $47.3 million. The second measurement record becomes:
support_relation = SUPPORTED provenance = VERIFIED epistemic_adequacy = ANSWERABLE The assertion policy now returns:
commitment = PERMIT next_action = NONE matched_rule = VERIFIED_RESPONSE The claim is authorized as a response.
Only after that does the same content become a memory candidate and face a separate admission policy. That policy reads inputs the assertion path never touched: how many independent source families corroborate the value, and whether the value carries a temporal-validity window.
cand_v2 fails both. The number rests on a single filing, and nothing records which period it describes.
MEMORY ADMISSION admission = QUARANTINE lifecycle = QUARANTINED allowed_uses = CONVERSATIONAL_CONTEXT, REGRESSION_TEST matched_rule = ANSWERABLE_BUT_NOT_DURABLE_FACT missing = SECOND_INDEPENDENT_SOURCE_FAMILY, VALID_TIME The same content, the same measurement record, two gates:
assertion : PERMIT VERIFIED_RESPONSE memory : QUARANTINE ANSWERABLE_BUT_NOT_DURABLE_FACT The system can tell the board the number. It cannot yet file the number away as a fact it will retrieve and trust later. CONVERSATIONAL_CONTEXT is allowed; FACTUAL_EVIDENCE is not.
This is the distinction the architecture must preserve:
response PERMIT ↓ eligible for memory evaluation ↓ separate memory admission decision not:
response PERMIT → automatically store as trusted fact The external send remains blocked throughout until a human-review artifact exists:
review_decision = { "review_id": "review_42", "decision": "APPROVE_ACTION", "scope": "SEND_BOARD_UPDATE", "reviewer_role": "finance_controller", "evidence_seen": ["evidence_v2"], "reason_codes": [ "Q3_VERIFIED", "BOARD_SEND_APPROVED", ], } That approval is a durable authorization artifact.
It is not automatically factual evidence.
If the reviewer separately verifies a proposition under a defined role and scope, that verification can be recorded as its own evidence-bearing artifact.
The action policy can then return:
authorization = PERMIT requirement = HUMAN_APPROVAL_SATISFIED and the enforcement point may execute the action.
Later, a second independent source — the audited earnings release — corroborates the value, and the reporting period is recorded:
cand_v3 independent_source_families = regulatory_filing, audited_earnings_release valid_time = 2024-Q3 Only now does the same number cross the persistence boundary:
MEMORY ADMISSION admission = ADMIT lifecycle = ACTIVE allowed_uses = CONVERSATIONAL_CONTEXT, FACTUAL_EVIDENCE matched_rule = CORROBORATED_TIME_BOUNDED_FACT One piece of content has therefore crossed three genuinely independent gates:
ASSERTION May this be stated? ACTION May this side effect execute? PERSISTENCE May this become durable state, and with which future-use capabilities? That is the architecture of the book in miniature.
Where we are The argument has accumulated one correction at a time:
hallucination taxonomy ↓ evidence model ↓ measurement contracts ↓ Hallucination Energy ↓ evaluation ↓ adversarial testing ↓ measurement boundaries ↓ consistency + sensitivity ↓ epistemic adequacy ↓ policy ↓ verification + repair ↓ memory governance The last task is synthesis.
The question is no longer:
How do we make the model trustworthy? It is:
How do we build a useful system whose consequential behavior does not depend on trusting every model output?
1. The system is the unit of reliability A modern AI application is rarely:
prompt → model → answer It is more often:
request ↓ retrieval ↓ model ↓ tools ↓ new observations ↓ more model calls ↓ policy ↓ actions ↓ memory ↓ future retrieval That is a compound system.
A strong model can sit inside a fragile system:
excellent model + unverified retrieval + auto-executed tools + append-only memory = fragile system A fallible model can sit inside a more governable system:
fallible model + controlled evidence + typed measurements + explicit policy + action mediation + versioned memory + replayable lineage = more governable system Model quality still matters.
But model benchmark accuracy and system reliability are not the same object.
Behavior-driven evaluation of compound AI systems makes the same general point from an evaluation perspective: aggregate benchmarks can miss failures that appear only under scenario-specific system behavior.[1]
So the final unit of analysis is:
THE COMPLETE PATH FROM INPUT TO CONSEQUENCE. 2. Distrust means separation of authority The word distrust does not mean:
assume every model output is false. It means:
do not give one probabilistic component unbounded authority over proposal, observation, verification, authorization, execution, and persistence. A model can be excellent at:
writing planning classification summarization repair proposals hypothesis generation query generation without being granted unilateral authority to decide:
what counts as established evidence what action executes what becomes durable state whether its own repair succeeded which hard policy requirement applies Distrust is not pessimism. It is separation of proposal from authority.
This is ordinary systems engineering.
A compiler does not decide whether deployment is authorized.
A database query builder does not define access policy.
A test runner does not decide whether a regulatory filing may be submitted.
The same discipline applies to LLMs.
3. There are three commitment boundaries The book’s terms for the decisions at these boundaries are consistent: commitment at the assertion boundary (Chapter 12), authorization at the action boundary, and memory admission at the persistence boundary. They are the same kind of decision — may this cross the gate — applied to three different things.
Assertion boundary May this candidate be presented as an assertion? Examples:
answer user publish report return API response include factual claim in document Action boundary May this proposed side effect execute? Examples:
send email write file merge pull request approve payment change database row invoke privileged tool Persistence boundary May this information become durable state, and which future uses may consume it? Examples:
memory profile vector store knowledge graph summary training example agent experience The same underlying content can receive:
assertion = PERMIT_WITH_QUALIFICATION action = HOLD_FOR_APPROVAL memory = PROVISIONAL_CONTEXT_ONLY That is not inconsistency.
It is typed authorization.
$$ \boxed{ \text{assertion permission} \neq \text{action permission} \neq \text{persistence permission} } $$ 4. The final architecture has three authority planes A two-way split between “probabilistic” and “deterministic” components is too coarse.
The book actually built three different kinds of authority.
Proposal plane What might we say or do?
generators planners repair models memory extractors query generators Observation / measurement plane What does the world, evidence, runtime, or candidate show?
retrieval tool observations source resolution entity/scope/time resolution validators classifiers detectors critics verifiers These components may still be probabilistic or fallible.
Their authority is diagnostic, not executive.
Control plane Which state transition is authorized?
policy budgets state machine action authorization enforcement memory admission revocation replay controls And beneath all three sits a durable state substrate:
evidence snapshots candidate lineage measurement records policy decisions action records memory events The important distinction is therefore not merely:
probabilistic vs deterministic. It is:
proposal authority observation authority transition authority. The three planes and their relationships are:
graph TD subgraph Proposal_Plane G[generators, planners, repair models, memory extractors, query generators] end subgraph Observation_Plane O[retrieval, tool observations, source resolution, validators, classifiers, detectors, critics, verifiers] end subgraph Control_Plane C[policy, budgets, state machine, action authorization, enforcement, memory admission, revocation, replay controls] end subgraph Durable_State S[evidence snapshots, candidate lineage, measurement records, policy decisions, action records, memory events] end Proposal_Plane --> Observation_Plane Observation_Plane --> Control_Plane Control_Plane --> Durable_State Durable_State --> Proposal_Plane Reliability comes from separating who may propose, who may observe, and who may authorize state transitions, with durable records closing the loop for later generations.
5. Deterministic arbitration is only as good as its inputs Where policy semantics are determinate, authorization should normally be deterministic and replayable.
That gives us:
review unit tests version control replay policy diff audit But deterministic arbitration is deterministic given its inputs.
If the system misclassifies:
risk tier user intent entity identity verification state requested action then perfect deterministic policy can enforce the wrong state perfectly.
So a bounded guarantee must name its assumptions.
For example:
Given validated policy inputs I, policy version P, and enforcement implementation F, action A cannot cross boundary B unless rule R is satisfied. That is very different from:
the AI is safe. Chapter 12’s warning remains:
Deterministic policy can be deterministically wrong.
The value of determinism is inspectability and enforceability, not omniscience.
6. The hardest objection: the stack checks itself with its own kind Here is the strongest argument against this book.
The architecture surrounds one fallible language model with a verification stack. But look at what the stack is made of. The claim extractor is a language model. The entailment check is a trained classifier. The support judge is a language model. The decision extractor, the memory-candidate extractor, the counterfactual-test oracle — models again. If the model at the center can be wrong, so can most of the components meant to catch it, and often in the same direction on the same inputs. Retrieval that misreads a query and a judge that misreads the same evidence are not independent failures.
So what is the reliability of a system whose verification stack is largely built from the same class of component it is verifying?
The honest answer is: partial, and it depends on which components carry the guarantee.
A few components are genuinely different in kind:
the runtime trace — the tool ran or it did not an authoritative database — the record says X or it does not a compiler, a test suite — the code builds or it does not a schema / type check — the structure is valid or it is not the deterministic policy engine — the rule fired or it did not the enforcement point — the side effect executed or it did not a scoped human verification — a person checked this claim, on the record These do not share the generator’s failure modes, because they are not doing language understanding. When a guarantee rests on one of them, it is a real guarantee, bounded by its own assumptions (instrumentation is complete, the database is current, the policy is correct).
Everything else — the extractors, the judges, the geometric detectors, the critics — reduces correlated risk without eliminating it. Stacking several imperfect model-based checks helps to the extent their errors are not perfectly aligned. Their blind spots do not completely coincide (Chapter 8 §21), so a claim that slips past containment may still be caught by entailment or by provenance. But the residual is a joint failure of correlated components, and it does not shrink to zero by adding more of the same kind.
This is why the boundaries in this chapter are drawn where they are. The assertion, action, and persistence gates, and the enforcement point, are deterministic and sit outside the model on purpose. The stochastic components feed them evidence; they do not hold the authority. The thesis is not “the surrounding components are reliable.” It is narrower and it survives this objection:
Put the guarantee on the components that do not share the generator’s failure modes. Use the rest to lower the odds, and measure how much.
7. Contain stochasticity before consequential commitment Not every component should be deterministic.
Generation, ranking, semantic retrieval, and some measurements gain their usefulness precisely from statistical generalization.
The design goal is not:
remove stochasticity. It is:
Contain stochasticity inside components whose outputs cannot directly create consequential commitment.
“Consequential” matters more than “irreversible.”
An email can sometimes be recalled.
A database write can sometimes be rolled back.
A memory can sometimes be revoked.
They are still consequential transitions and deserve mediation.
8. Every consequential boundary should consume typed artifacts A system becomes fragile when semantic distinctions live only in naming conventions and prose.
Prefer structured artifacts.
A candidate:
candidate = { "candidate_id": "cand_v7", "parent_id": "cand_v6", "claims": ["c1", "c2", "c3"], "evidence_snapshot": "evidence://bundle/1842", "generator_version": "generator-v17", } A reliability record:
reliability = { "containment": {...}, "structural_fidelity": {...}, "sensitivity": {...}, "epistemic_adequacy": {...}, "provenance": {...}, } A policy decision:
policy_decision = { "commitment": "HOLD", "next_action": "VERIFY", "obligations": ["VERIFY_PROVENANCE"], "matched_rule": "PROVENANCE_REQUIRED", } A human decision:
human_decision = { "decision": "APPROVE_ACTION", "scope": "SEND_BOARD_UPDATE", "reviewer_role": "finance_controller", "evidence_seen": ["evidence_v2"], } A memory record:
origin verification valid time system time lifecycle claim-level lineage allowed uses This prevents accidental semantics such as:
null means safe absence means pass retrieved means trusted stored means true PERMIT means persist The architecture preserves distinctions in its types.
9. Evidence enters before authority The evidence layer still begins with resolved objects:
claim proposition source evidence item support relation provenance freshness conflict A useful path is:
SOURCE / TOOL / USER INPUT ↓ NORMALIZE ↓ RESOLVE IDENTITY + SCOPE + TIME ↓ ATTACH PROVENANCE ↓ EVIDENCE SNAPSHOT ↓ OBSERVATION / MEASUREMENT If the decision matters, preserve enough state to answer later:
Which source version? Which support span? Which retrieval result? Which resolver version? Which timestamp? The rest of the architecture cannot recover distinctions discarded here.
10. Measurements remain sensors, not permissions The book accumulated several possible measurement families:
containment structural fidelity consistency sensitivity answerability / epistemic adequacy provenance runtime/tool checks No application needs all of them for every request.
But every deployed sensor needs a contract:
What property is targeted? What observable is measured? What assumptions connect them? What are the blind spots? What does PASS mean? A containment signal cannot average away:
provenance = FAIL A fluent answer cannot average away:
epistemic_adequacy = INSUFFICIENT_EVIDENCE A context-sensitive answer cannot average away:
policy_restriction = TRUE Never allow an average to cancel a non-negotiable failure.
11. Policy selects among feasible transitions Policy does two things.
First, hard requirements eliminate impermissible transitions.
Conceptually:
$$ \mathcal A_{feasible}(\mathcal R,c,h) $$contains only actions that satisfy non-negotiable constraints.
Then the system can optimize among feasible actions for:
latency cost coverage human burden user interruption So:
hard constraints ↓ feasible transitions ↓ optimize utility within feasible set not:
average every risk and utility term together. The policy function remains stateful:
$$ \pi(\mathcal R,c,h) \rightarrow (\text{commitment},\text{next action},\text{obligations}). $$Execution history matters:
first evidence gap → RETRIEVE same gap after unsuccessful retrieval → REVIEW budget exhausted → ABSTAIN Runtime-governance work increasingly treats partial execution paths as part of the policy input rather than governing only the latest prompt.[3]
12. Enforcement must sit outside the model A system prompt can say:
Do not send emails without approval. That is steering.
It is not enforcement.
The stronger pattern is:
MODEL proposes tool call ↓ POLICY DECISION ↓ POLICY ENFORCEMENT POINT ↓ execute / block / escalate A minimal enforcement function is conceptually:
def enforce(tool_call, decision, approvals): if decision.authorization != "PERMIT": raise AuthorizationError("tool execution blocked") if decision.requires_human_approval: if tool_call.capability not in approvals: raise AuthorizationError("human approval missing") return execute(tool_call) The executor consumes typed authorization.
It does not ask the model whether its own explanation sounds acceptable.
Recent tool-agent work demonstrates this general architecture by placing policy enforcement at the tool-call boundary rather than relying solely on prompts.[4]
Governance-by-construction work similarly inserts controls at multiple agent checkpoints.[5]
Prompt instructions shape proposals. Enforcement controls side effects.
13. Human review produces lineage, not magic truth Humans are another governed resource.
They have:
latency cost fatigue limited context inconsistent judgment capacity constraints A reviewer should receive:
candidate evidence failed measurements policy reason repair history lineage proposed action and return a typed decision.
An approval event can authorize an action within a declared scope.
It does not automatically establish every factual proposition present in the artifact.
If a reviewer performs explicit factual verification, record that as a separate verification artifact with:
reviewer role claim scope evidence seen verification decision time policy/version context That allows human intervention to participate in replayable lineage without turning humans into invisible stateless oracles.
14. Recovery is ordinary control flow HOLD is not an exception.
It is a state.
HOLD ↓ LOCALIZE FAILURE ↓ RETRIEVE / VERIFY / REFINE / ASK ↓ NEW CANDIDATE OR EVIDENCE STATE ↓ INVALIDATE DEPENDENCIES ↓ RE-MEASURE ↓ POLICY Recovery can terminate through:
PERMIT DENY ABSTAIN REVIEW NO_VALID_RECOVERY budget exhaustion cycle detection Every repair creates a new candidate state.
Every evidence update creates a new evidence state.
Repeated attempts are not evidence that the next attempt is correct.
15. Persistence is its own state-changing subsystem Memory candidates can originate from many places:
user assertions tool observations external sources model candidates agent trajectories summaries failure artifacts human-reviewed state They do not originate only from user-facing responses.
The write path is:
graph TD US[USER / TOOL / SOURCE / MODEL / EXPERIENCE / FAILURE] --> MC[MEMORY CANDIDATE] MC --> OVT[ORIGIN + VERIFICATION + TIME + LINEAGE + CAPABILITIES] OVT --> MA[MEMORY ADMISSION POLICY] MA --> PERS[PERSISTED WITH USE RIGHTS] The write path treats persistence as a governed capability grant, not as a side effect of producing or observing text.
The read path is:
graph TD Q[QUERY + PURPOSE + CALLER] --> AE[ACCESS ELIGIBILITY] AE --> RET[RETRIEVAL] RET --> EA[EVIDENTIAL ADMISSIBILITY] EA --> UC[USABLE CONTEXT] And the result loops back into proposal and measurement.
Chapter 14’s rule survives unchanged:
$$ \text{response PERMIT} \not\Rightarrow \text{memory ADMIT}. $$ Model output must never become evidence merely because the system remembered it.
16. The ledger supports several different kinds of replay The word replay hides different operations.
Exact replay Run an immutable historical artifact through a deterministic component using the same inputs and version.
Example:
measurement record + policy-v2 → deterministic policy decision Counterfactual re-evaluation Freeze the historical candidate and evidence, but intentionally change one component:
same candidate + evidence policy-v1 → policy-v2 or:
same candidate + evidence detector-v4 → detector-v5 This isolates the effect of the changed component.
Full-stack re-execution Run the original request and captured external observations through the current system.
This may not exactly reproduce the old trajectory if:
generation is stochastic hosted model implementation changed seed control is unavailable external tool behavior cannot be reconstructed So the ledger should be honest about capability:
EXACT_REPLAY_AVAILABLE = false COUNTERFACTUAL_REEVALUATION_AVAILABLE = true FULL_STACK_REEXECUTION_AVAILABLE = true For meaningful analysis, preserve where possible:
request candidate versions model IDs/configuration random seed where meaningful tool requests tool responses retrieval results source versions evidence snapshots measurement versions policy versions human decisions memory writes side effects And replay/re-execution must be dry-run by default:
no real email no real payment no memory write no external mutation The same enforcement logic should run, but side effects become recorded simulations.
17. Sensor disagreement and missing measurements are useful state Suppose:
containment = PASS structural_fidelity = PASS sensitivity = FAIL answerability = ANSWERABLE provenance = VERIFIED The wrong move is:
average → 0.81 → ACCEPT The disagreement is the diagnosis:
factual support appears adequate but the recommendation ignores decisive context. Conversely:
containment = FAIL support verifier = PASS provenance = VERIFIED may expose a sensor limitation.
Likewise, missing checks remain explicit:
UNAVAILABLE NOT_MEASURED NOT_RUN UNCERTAIN If progressive evaluation terminates early, record why:
state = NOT_RUN reason = EARLY_TERMINATION_POLICY_BLOCK not merely:
missing. The application policy decides what these states imply.
The generator does not.
18. Evaluate paths and attribute the first failed boundary After fifteen chapters, end-to-end correctness cannot mean only:
Did the final sentence look right? A system can fail in:
retrieval resolution measurement verification policy enforcement repair memory human review A behavior specification can say:
GIVEN Q3 revenue is absent from admissible evidence WHEN the model proposes a numerical Q3 value THEN assertion commitment = HOLD next action = RETRIEVE memory admission = QUARANTINE external send = BLOCKED A paired case adds authoritative Q3 evidence:
THEN supported assertion may be PERMITTED memory may be separately ADMITTED external send still follows action policy System-level failures should also record:
first_invalid_state first_failed_boundary propagated_failures final_consequence For example:
FIRST FAILURE retrieval returned stale evidence DOWNSTREAM support verifier correctly evaluated the stale snapshot policy correctly acted on its input state CONSEQUENCE incorrect assertion emitted That separates root cause from downstream propagation.
Bhagat et al.’s compound-system framework similarly argues for explicit behavior specifications and scenario tests rather than aggregate benchmark scores alone.[1]
19. Test the boundaries directly The most important tests are often stronger than language-quality tests.
Assertion unestablished claim → cannot be emitted as verified fact Action prohibited tool proposal → cannot execute even if model insists Enforcement policy = DENY executor receives tool call anyway → side effect still impossible Persistence HOLD candidate → cannot become active factual memory Lineage unverified ancestor → summarization cannot erase obligation Recovery repair → no inherited authorization Unknown state NOT_MEASURED → never silently treated as PASS Budget retry limit reached → deterministic terminal route These are boundary invariants.
They are often easier to establish strongly than the model’s internal reasoning.
20. Reliability is a consequence tax, not a blanket setting A full Level-7 reliability path is wasteful for:
Write a limerick about a dog. The same shortcut can be reckless for:
Send a financial update to the board. So the control plane should scale with consequence.
A simple risk router can consider:
factual consequence action blast radius persistence duration reversibility privacy sensitivity financial/legal exposure human impact Then choose a path.
Fast path Suitable for low-consequence tasks:
basic policy restrictions cheap deterministic checks generation Governed path For higher consequence:
source/provenance checks typed measurements external verification strong action mediation memory admission human review where required This is the consequence tax:
The more expensive a failure is, the more reliability work the system should be willing to buy before commitment.
This does not mean every expensive check belongs at the end.
Progressive evaluation should order work using both:
blocker severity and expected information gained per unit cost/latency Hard cheap blockers go early.
Expensive sensitivity suites and human review can be reserved for cases where they can still change the decision.
Measure the resulting path against:
P50/P95/P99 latency API cost human-review capacity coverage false acceptance false rejection 21. Monitor the control plane, not only the model Operational observability should track the distribution of decisions.
Suppose historically:
RETRIEVE = 2% REVIEW = 0.5% ABSTAIN = 1% and suddenly:
RETRIEVE = 45%. That is not automatically evidence that the system became safer.
Possible causes include:
vector database outage embedding model change knowledge-base deletion provenance resolver bug policy change source staleness spike Monitor distributions such as:
PERMIT / HOLD / DENY RETRIEVE / VERIFY / ASK / REVIEW human approval rate memory ADMIT / QUARANTINE NOT_RUN / UNAVAILABLE matched-rule frequency first-failed-boundary frequency Alert on meaningful shifts.
The control plane can drift operationally even when no model version changes.
22. Practical adoption is incremental—but not monotonically safer A minimum useful architecture is:
1. version consequential candidates 2. preserve evidence and provenance 3. use explicit typed state at every consequential boundary 4. separate policy from generation 5. mediate consequential tools 6. gate persistent memory 7. retain enough lineage for replay 8. turn incidents into regression tests A maturity ladder can then guide implementation.
Level 0 — Raw generation prompt → model → output Level 1 — Grounded generation retrieval → model Exit test:
Can we identify exactly which evidence entered the request? Level 2 — Measured generation candidate → typed sensors Exit test:
Can every required measurement report explicit state, including NOT_MEASURED / UNAVAILABLE? Level 3 — Policy-mediated output reliability record → policy Exit test:
Can we replay a policy version and explain the decision diff? Level 4 — Governed actions tool proposal → external enforcement Exit test:
Can we prove a prohibited tool call cannot execute even when the model proposes it? Level 5 — Bounded recovery verify → repair → re-measure → re-authorize Exit test:
Does every recovery loop terminate under budget/cycle rules? Level 6 — Governed memory write/read admission + lineage + revocation Exit test:
Can a HOLD/model-derived candidate be prevented from becoming active factual memory? Level 7 — Replayable reliability engineering behavior tests policy replay measurement replay incident regression control-plane monitoring This is a capability/governance ladder, not a theorem that each added subsystem is automatically safer.
Level 0 → 1 is the clearest example. Adding retrieval gives the model grounding, but it also opens an attack surface that raw generation did not have: a document in the corpus can now carry an injected instruction or a planted false fact straight into the context. A Level-1 system with an unfiltered retriever can be less safe on adversarial inputs than the Level-0 system it replaced. The same holds at Level 5 → 6 — persistence is the channel Chapter 14 is entirely about.
So the ladder measures capability, not safety. Each rung adds a subsystem, and each subsystem has failure modes that must be evaluated in its own right. Do not read a level number as a safety rating.
23. The complete architecture is a closed governed system The main control loop is:
graph TD R[REQUEST] --> CR[CONTEXT + RISK PROFILE] CR --> ER[EVIDENCE / MEMORY READ PATH] CR --> PP[PROPOSAL PLANE] ER --> OR[OBSERVATION RECORD] PP --> VC[VERSIONED CANDIDATE] OR --> OM[OBSERVATION / MEASUREMENT] VC --> OM OM --> RR[RELIABILITY RECORD] RR --> CP[CONTROL PLANE] CP --> HOLD[HOLD] CP --> PER[PERMIT] CP --> DEN[DENY] PER --> AG[ASSERTION GATE] PER --> ACT[ACTION GATE] AG --> UA[USER/API] ACT --> EN[ENFORCEMENT] EN --> TR[TOOL RESULT] TR --> NO[NEW OBSERVATION] NO --> OM HOLD --> RL[RECOVERY LOOP] RL --> OM DEN --> AB[ABSTAIN / REJECT] The closed loop keeps requests, evidence, candidates, measurements, policy decisions, enforcement, and recovery in one governed runtime process.
Persistence is a separate state-changing loop:
graph TD US[USER / TOOL / SOURCE / MODEL / EXPERIENCE / FAILURE] --> MC[MEMORY CANDIDATE] MC --> PPOL[PERSISTENCE POLICY] PPOL --> PM[PERSISTED WITH USE CAPABILITIES] PM --> FS[FUTURE STATE] FS --> FRP[future read path] Running alongside both loops is the reliability ledger:
EVIDENCE SNAPSHOTS CANDIDATE LINEAGE MEASUREMENT RECORDS POLICY DECISIONS HUMAN DECISIONS ACTION EXECUTION MEMORY EVENTS VERSION HISTORY The architecture is therefore not a terminal pipeline.
Actions create observations.
Memory creates future input.
Repair creates new candidates.
Policy changes future paths.
The whole system is a governed feedback process.
A useful compression is:
$$ \boxed{ \text{proposal} \rightarrow \text{observation} \rightarrow \text{measurement} \rightarrow \text{authorization} \rightarrow \text{enforcement} \rightarrow \text{state} } $$with every state-changing edge capable of becoming future input.
24. AI can help design the system without owning the floor AI is useful throughout this architecture:
candidate generation claim decomposition memory proposal extraction repair proposals retrieval query generation counterfactual tests failure clustering policy-test generation human-review summarization But a dangerous shortcut remains:
Give the reliability JSON to a cheap LLM and ask whether to ACCEPT. That puts the policy floor back inside the probabilistic component.
A similarly dangerous generated policy is:
if record["state"] != "FAIL": allow() which silently turns:
NOT_MEASURED UNAVAILABLE UNCERTAIN into permission.
The model can propose rules.
The rules should then be translated into explicit executable semantics and tested.
Where requirements can be expressed mechanically:
verified provenance required human approval required retrieval attempts <= 2 FACTUAL_EVIDENCE capability forbidden keep the final enforcement floor outside unconstrained model judgment.
25. Ten laws—and where they came from Law 1 — Generation is not acceptance Generation may be stochastic. Acceptance does not have to be.
From Chapters 1 and 12.
Law 2 — A sensor is not a verdict measurement ≠ authorization From Chapters 4 and 6.
Law 3 — Preserve distinctions before scoring If a decision needs:
polarity role quantity time provenance preserve them until after the decision.
From Chapter 8.
Law 4 — Different failures need different recovery missing user input → ASK missing evidence → RETRIEVE conflict → VERIFY bad candidate → REFINE unrecoverable → ABSTAIN / REJECT From Chapters 11–13.
Law 5 — Every repair creates a new state repair → re-measure → re-authorize From Chapter 13.
Law 6 — Relevance is not admissibility A retrieved item can be highly relevant and still be forbidden as evidence.
From Chapter 14.
Law 7 — Transformation does not manufacture evidence Summarization, rephrasing, repetition, storage, and retrieval do not by themselves create independent factual support.
From Chapters 13–14.
Law 8 — Side effects need external mediation The model proposes actions.
A trusted runtime decides whether they execute.
From Chapters 12 and 15.
Law 9 — Persistent state needs its own gate response PERMIT ≠ memory ADMIT From Chapter 14.
Law 10 — Reliability claims must be scoped Never reduce the conclusion to:
this model is trustworthy. State:
workload policy version measurements evidence assumptions enforcement boundary observed failure regime From Chapters 6, 12, and 15.
26. What the book removed The book’s progression can be read as a sequence of hidden assumptions that no longer survive.
Chapter Assumption removed 1 Fluency is verification 2 Hallucination is one failure class 3 Evidence, truth, support, provenance and verification are interchangeable 4 A metric is a verdict 5 Semantic containment is proof of truth 6 Detector quality equals deployment utility 7 Average benchmark performance is enough 8 Lost structure can be recovered from a scalar 9 Reliability means invariance to every change 10 Safe-looking generic output is useful reasoning 11 Always answering is a capability 12 Measurements can authorize themselves 13 A repair inherits trust 14 Stored or repeated text becomes evidence 15 The model is the system The architecture is what remains once those shortcuts are removed.
27. What you should now be able to answer Why is assertion PERMIT not enough to send an email? Because assertion and action are different commitment boundaries. The first authorizes a content claim to be emitted. The second authorizes a side effect. An externally consequential action can still require human approval, capability checks, or other obligations after the response itself is factually authorized.
Why is response PERMIT not enough to store factual memory? Because memory is future state. Persistence requires its own admission policy over origin, verification, time, lineage, and future-use capabilities. A permitted response is only eligible to become a memory candidate.
Why can deterministic policy still produce a bad outcome? Because policy is deterministic given its inputs. If risk classification, entity resolution, verification, or another policy input is wrong, deterministic arbitration can consistently authorize the wrong transition. Reliability claims must therefore state both policy semantics and input assumptions.
Why is a tool result not simply the end of an action? Because tool execution changes or observes the world. The result becomes a new observation that can alter evidence, invalidate measurements, change the candidate, and trigger another policy decision. The architecture is a feedback system, not a one-way pipeline.
Why is control-plane distribution monitoring a reliability concern? Because sudden changes in HOLD, RETRIEVE, REVIEW, or QUARANTINE rates can reveal failures in retrieval, evidence stores, policy configuration, or verification even when the model has not changed. Policy decisions are operational signals.
What is the strongest reliability claim this architecture supports? A bounded claim such as:
Under policy P, validated inputs I, evidence assumptions E, and enforcement implementation F, action A cannot cross boundary B unless rule R is satisfied. It does not support the global statement:
the AI is safe. 28. Deployment questions Before deploying a consequential workflow, the boundary tests in Section 19 must pass: a prohibited tool call cannot execute, a HOLD candidate cannot become factual memory, an unverified ancestor’s obligation survives summarization, NOT_MEASURED is never read as PASS, and every recovery loop terminates.
Then ask the questions the boundary tests do not cover:
Evidence Can we reconstruct the evidence that existed at decision time? Candidate Are consequential outputs versioned rather than invisibly mutated? Policy Which constraints are hard blockers, and which transitions remain feasible? Human review Is approval scoped, typed, and recorded with the evidence the reviewer saw? Replay Which decisions are exactly replayable, counterfactually re-evaluable, or only re-executable? Operations Are policy-decision and route distributions monitored for drift? Evaluation Do behavior tests exercise complete paths and record the first failed boundary? If these have no explicit answers, the application is still depending on invisible trust.
29. Exercises Exercise 1 — Build the three boundaries Create a factual assistant that can also send email and store memory.
Implement separate:
assertion_authorization action_authorization memory_admission Construct one case in which all three return different outcomes.
Exercise 2 — Test the enforcement point Make the model propose:
SEND_EMAIL while policy returns:
DENY Attempt to bypass the policy layer and send the call directly to the executor.
The enforcement point should still prevent the side effect.
Exercise 3 — Whole-system counterfactual Create:
A: decisive evidence absent B: decisive evidence present Expected behavior:
A → HOLD + RETRIEVE / ABSTAIN B → assertion may PERMIT Evaluate assertion, action, and persistence separately.
Exercise 4 — Replay without side effects Store:
candidate evidence snapshot measurement record policy-v1 decision Create policy-v2 with one stricter provenance requirement.
Re-evaluate historical cases in dry-run mode.
Confirm that no external tool or memory write executes during replay.
Exercise 5 — Attribute the first failed boundary Construct an end-to-end failure caused by stale retrieval.
Record:
first_invalid_state first_failed_boundary propagated_failures final_consequence Verify that a later policy decision is not incorrectly blamed for a failure that originated upstream.
Exercise 6 — Monitor policy drift Generate a baseline distribution over:
PERMIT HOLD RETRIEVE VERIFY REVIEW QUARANTINE Then simulate a retrieval outage.
Measure which routing distributions move and design an alert.
30. The deeper lesson Return to the fictional Northbridge Medal from Chapter 1.
The model that could fluently invent an answer there can sit inside this final architecture unchanged.
It may still invent.
The difference is that the invention no longer receives invisible authority merely because it was fluent.
It can be:
measured held verified rejected repaired prevented from executing prevented from persisting without requiring the generator itself to become infallible.
That is the systems move.
A language model is extraordinarily useful precisely because it can generate beyond what has been explicitly programmed.
The same property means it should not be treated as unquestioned authority over everything it generates.
Build systems in which:
a model can be wrong without the database becoming wrong; a model can propose a bad action without the action executing; a model can generate an unsupported claim without the claim becoming future evidence; a verifier can be uncertain without uncertainty silently becoming permission; a repair can fail without the repaired answer inheriting trust; a policy can change without losing the ability to re-evaluate old decisions. The final loop is:
MODEL PROPOSES ↓ SYSTEM OBSERVES ↓ SYSTEM MEASURES ↓ SYSTEM DECIDES ↓ SYSTEM ENFORCES ↓ SYSTEM RECORDS ↓ INCIDENTS BECOME TESTS The model remains at the center of the application.
It is no longer the center of authority.
The reliable unit is not the model. It is the system around the model.
Its reliability no longer depends on pretending the model is infallible.
It can instead earn bounded reliability under explicit assumptions while the model remains fallible.
Research roots Pranav Bhagat, K N Ajay Shastry, Pranoy Panda and Chaitanya Devaguptapu, “Evaluating Compound AI Systems through Behaviors, Not Benchmarks,” Findings of EMNLP 2025, pp. 24193–24222. Proposes behavior-driven evaluation specifications for compound AI systems and reports that scenario-oriented tests expose failures missed by conventional benchmark evaluation. https://aclanthology.org/2025.findings-emnlp.1314/
C. Brian Smith and Daniel McCarthy, “Deterministic governance for generative systems: a policy-aligned runtime for validating AI outputs,” AI and Ethics 6, article 394, 2026. Describes a bounded deterministic arbitration architecture with versioned constraints and evidence records for replayable governance decisions. https://doi.org/10.1007/s43681-026-01172-6
Maurits Kaptein, Vassilis-Javed Khan and Andriy Podstavnychy, “Runtime Governance for AI Agents: Policies on Paths,” arXiv:2603.16586, 2026. Formalizes runtime governance over partial execution paths and proposed next actions, emphasizing that path-dependent behavior cannot be governed solely by static prompts or static access control. https://arxiv.org/abs/2603.16586
Shanshan Wang, Sizheng Zhu and Rende Li, “Runtime Policy Enforcement for MCP-Based LLM Agents,” Electronics 15(13), 2829, 2026. Implements a policy enforcement point at the tool-call boundary with declarative rules, cross-step information-flow labels, and audit logging. https://doi.org/10.3390/electronics15132829
Segev Shlomov, Iftach Shoham, Alon Oved, Ido Levy, Sami Marreed, Harold J. Ship, Offer Akrabi, Sergey Zeltyn, Avi Yaeli and Nir Mashkif, “Governance by Construction for Generalist Agents,” Proceedings of the ACM Conference on AI and Agentic Systems, 2026, pp. 1280–1287. Demonstrates policy-as-code interventions across multiple agent execution checkpoints, including intent, tool use, human approval and output. https://doi.org/10.1145/3786335.3813192
Chloe Autio, Reva Schwartz, Jesse Dunietz, Shomik Jain, Martin Stanley, Elham Tabassi, Patrick Hall and Kamie Roberts, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1, 2024, updated 2026. Frames generative-AI trustworthiness as lifecycle risk management involving governance, measurement, management and ongoing evaluation. https://doi.org/10.6028/NIST.AI.600-1
End The book began with hallucination as a model behavior.
It ends with reliability as a systems property.
That is the shift.