Building Systems That Distrust Their Models
The first chapter began with a simple observation:
A language model can produce a fluent answer
without possessing a mechanism that proves the answer is true.
Fourteen chapters later, that fact has not changed.
The model can still:
invent
misbind
misattribute
ignore decisive context
answer without enough evidence
accept bad retrieval
repair one error by creating another
repeat its own stored mistake
The final architecture does not make those possibilities disappear.
It changes who has authority over what happens next.
The model proposes. The system decides what survives.
That is the final systems principle of this book.
Environment and canonical reference implementation
The end-to-end example in this chapter is backed by a small deterministic controller:
Environment: Python 3.13.5
Dependencies: Python standard library only
Canonical implementation:
experiments/ch15/distrustful_controller.py
Run it with:
python experiments/ch15/distrustful_controller.py
The script is not a production framework.
Its purpose is narrower: demonstrate the boundary invariants in executable form.
A complete run through the system
Consider one task:
Prepare a Q3 revenue update for the board.
The initial evidence snapshot contains:
Q1 revenue = $41.2M
Q2 revenue = $43.8M
A model proposes:
Q3 revenue was approximately $46 million.
The important point is not that $46M is known to be false.
The current evidence simply does not establish it.
The first measurement record is therefore:
CANDIDATE
cand_v1
MEASUREMENT
support_relation = NOT_ESTABLISHED
provenance = UNVERIFIED
epistemic_adequacy = INSUFFICIENT_EVIDENCE
The assertion policy returns:
ASSERTION POLICY
commitment = HOLD
next_action = RETRIEVE
matched_rule = EVIDENCE_REQUIRED
The candidate separately reaches the persistence boundary:
MEMORY ADMISSION
admission = QUARANTINE
lifecycle = QUARANTINED
allowed_uses = DEBUG, REGRESSION_TEST
Nothing in the response decision automatically admits the claim to factual memory.
Now suppose the model proposes a side effect:
SEND_BOARD_UPDATE
The action policy independently returns:
ACTION POLICY
authorization = HOLD
requirement = HUMAN_APPROVAL_REQUIRED
The enforcement point does not care how persuasive the model sounds.
The reference controller actually raises:
AuthorizationError: action SEND_BOARD_UPDATE requires HUMAN_APPROVAL_REQUIRED; execution blocked
No email is sent.
Now retrieval obtains an authoritative Q3 filing:
Q3 revenue = $47.3M
The system creates a new evidence snapshot and a new candidate:
cand_v2
Q3 revenue was $47.3 million.
The second measurement record becomes:
support_relation = SUPPORTED
provenance = VERIFIED
epistemic_adequacy = ANSWERABLE
The assertion policy now returns:
commitment = PERMIT
next_action = NONE
matched_rule = VERIFIED_RESPONSE
The claim is authorized as a response.
Only after that does the same content become a memory candidate and face a separate admission policy. That policy reads inputs the assertion path never touched: how many independent source families corroborate the value, and whether the value carries a temporal-validity window.
cand_v2 fails both. The number rests on a single filing, and nothing records which period it describes.
MEMORY ADMISSION
admission = QUARANTINE
lifecycle = QUARANTINED
allowed_uses = CONVERSATIONAL_CONTEXT, REGRESSION_TEST
matched_rule = ANSWERABLE_BUT_NOT_DURABLE_FACT
missing = SECOND_INDEPENDENT_SOURCE_FAMILY, VALID_TIME
The same content, the same measurement record, two gates:
assertion : PERMIT VERIFIED_RESPONSE
memory : QUARANTINE ANSWERABLE_BUT_NOT_DURABLE_FACT
The system can tell the board the number. It cannot yet file the number away as a fact it will retrieve and trust later. CONVERSATIONAL_CONTEXT is allowed; FACTUAL_EVIDENCE is not.
This is the distinction the architecture must preserve:
response PERMIT
↓
eligible for memory evaluation
↓
separate memory admission decision
not:
response PERMIT
→ automatically store as trusted fact
The external send remains blocked throughout until a human-review artifact exists:
review_decision = {
"review_id": "review_42",
"decision": "APPROVE_ACTION",
"scope": "SEND_BOARD_UPDATE",
"reviewer_role": "finance_controller",
"evidence_seen": ["evidence_v2"],
"reason_codes": [
"Q3_VERIFIED",
"BOARD_SEND_APPROVED",
],
}
That approval is a durable authorization artifact.
It is not automatically factual evidence.
If the reviewer separately verifies a proposition under a defined role and scope, that verification can be recorded as its own evidence-bearing artifact.
The action policy can then return:
authorization = PERMIT
requirement = HUMAN_APPROVAL_SATISFIED
and the enforcement point may execute the action.
Later, a second independent source — the audited earnings release — corroborates the value, and the reporting period is recorded:
cand_v3
independent_source_families = regulatory_filing, audited_earnings_release
valid_time = 2024-Q3
Only now does the same number cross the persistence boundary:
MEMORY ADMISSION
admission = ADMIT
lifecycle = ACTIVE
allowed_uses = CONVERSATIONAL_CONTEXT, FACTUAL_EVIDENCE
matched_rule = CORROBORATED_TIME_BOUNDED_FACT
One piece of content has therefore crossed three genuinely independent gates:
ASSERTION
May this be stated?
ACTION
May this side effect execute?
PERSISTENCE
May this become durable state,
and with which future-use capabilities?
That is the architecture of the book in miniature.
Where we are
The argument has accumulated one correction at a time:
hallucination taxonomy
↓
evidence model
↓
measurement contracts
↓
Hallucination Energy
↓
evaluation
↓
adversarial testing
↓
measurement boundaries
↓
consistency + sensitivity
↓
epistemic adequacy
↓
policy
↓
verification + repair
↓
memory governance
The last task is synthesis.
The question is no longer:
How do we make the model trustworthy?
It is:
How do we build a useful system whose consequential behavior does not depend on trusting every model output?
1. The system is the unit of reliability
A modern AI application is rarely:
prompt → model → answer
It is more often:
request
↓
retrieval
↓
model
↓
tools
↓
new observations
↓
more model calls
↓
policy
↓
actions
↓
memory
↓
future retrieval
That is a compound system.
A strong model can sit inside a fragile system:
excellent model
+
unverified retrieval
+
auto-executed tools
+
append-only memory
=
fragile system
A fallible model can sit inside a more governable system:
fallible model
+
controlled evidence
+
typed measurements
+
explicit policy
+
action mediation
+
versioned memory
+
replayable lineage
=
more governable system
Model quality still matters.
But model benchmark accuracy and system reliability are not the same object.
Behavior-driven evaluation of compound AI systems makes the same general point from an evaluation perspective: aggregate benchmarks can miss failures that appear only under scenario-specific system behavior.[1]
So the final unit of analysis is:
THE COMPLETE PATH FROM INPUT TO CONSEQUENCE.
2. Distrust means separation of authority
The word distrust does not mean:
assume every model output is false.
It means:
do not give one probabilistic component
unbounded authority over proposal,
observation,
verification,
authorization,
execution,
and persistence.
A model can be excellent at:
writing
planning
classification
summarization
repair proposals
hypothesis generation
query generation
without being granted unilateral authority to decide:
what counts as established evidence
what action executes
what becomes durable state
whether its own repair succeeded
which hard policy requirement applies
Distrust is not pessimism. It is separation of proposal from authority.
This is ordinary systems engineering.
A compiler does not decide whether deployment is authorized.
A database query builder does not define access policy.
A test runner does not decide whether a regulatory filing may be submitted.
The same discipline applies to LLMs.
3. There are three commitment boundaries
The book’s terms for the decisions at these boundaries are consistent: commitment at the assertion boundary (Chapter 12), authorization at the action boundary, and memory admission at the persistence boundary. They are the same kind of decision — may this cross the gate — applied to three different things.
Assertion boundary
May this candidate be presented as an assertion?
Examples:
answer user
publish report
return API response
include factual claim in document
Action boundary
May this proposed side effect execute?
Examples:
send email
write file
merge pull request
approve payment
change database row
invoke privileged tool
Persistence boundary
May this information become durable state,
and which future uses may consume it?
Examples:
memory
profile
vector store
knowledge graph
summary
training example
agent experience
The same underlying content can receive:
assertion = PERMIT_WITH_QUALIFICATION
action = HOLD_FOR_APPROVAL
memory = PROVISIONAL_CONTEXT_ONLY
That is not inconsistency.
It is typed authorization.
4. The final architecture has three authority planes
A two-way split between “probabilistic” and “deterministic” components is too coarse.
The book actually built three different kinds of authority.
Proposal plane
What might we say or do?
generators
planners
repair models
memory extractors
query generators
Observation / measurement plane
What does the world, evidence, runtime, or candidate show?
retrieval
tool observations
source resolution
entity/scope/time resolution
validators
classifiers
detectors
critics
verifiers
These components may still be probabilistic or fallible.
Their authority is diagnostic, not executive.
Control plane
Which state transition is authorized?
policy
budgets
state machine
action authorization
enforcement
memory admission
revocation
replay controls
And beneath all three sits a durable state substrate:
evidence snapshots
candidate lineage
measurement records
policy decisions
action records
memory events
The important distinction is therefore not merely:
probabilistic
vs
deterministic.
It is:
proposal authority
observation authority
transition authority.
The three planes and their relationships are:
graph TD
subgraph Proposal_Plane
G[generators, planners, repair models, memory extractors, query generators]
end
subgraph Observation_Plane
O[retrieval, tool observations, source resolution, validators, classifiers, detectors, critics, verifiers]
end
subgraph Control_Plane
C[policy, budgets, state machine, action authorization, enforcement, memory admission, revocation, replay controls]
end
subgraph Durable_State
S[evidence snapshots, candidate lineage, measurement records, policy decisions, action records, memory events]
end
Proposal_Plane --> Observation_Plane
Observation_Plane --> Control_Plane
Control_Plane --> Durable_State
Durable_State --> Proposal_Plane
Reliability comes from separating who may propose, who may observe, and who may authorize state transitions, with durable records closing the loop for later generations.
5. Deterministic arbitration is only as good as its inputs
Where policy semantics are determinate, authorization should normally be deterministic and replayable.
That gives us:
review
unit tests
version control
replay
policy diff
audit
But deterministic arbitration is deterministic given its inputs.
If the system misclassifies:
risk tier
user intent
entity identity
verification state
requested action
then perfect deterministic policy can enforce the wrong state perfectly.
So a bounded guarantee must name its assumptions.
For example:
Given validated policy inputs I,
policy version P,
and enforcement implementation F,
action A cannot cross boundary B
unless rule R is satisfied.
That is very different from:
the AI is safe.
Chapter 12’s warning remains:
Deterministic policy can be deterministically wrong.
The value of determinism is inspectability and enforceability, not omniscience.
6. The hardest objection: the stack checks itself with its own kind
Here is the strongest argument against this book.
The architecture surrounds one fallible language model with a verification stack. But look at what the stack is made of. The claim extractor is a language model. The entailment check is a trained classifier. The support judge is a language model. The decision extractor, the memory-candidate extractor, the counterfactual-test oracle — models again. If the model at the center can be wrong, so can most of the components meant to catch it, and often in the same direction on the same inputs. Retrieval that misreads a query and a judge that misreads the same evidence are not independent failures.
So what is the reliability of a system whose verification stack is largely built from the same class of component it is verifying?
The honest answer is: partial, and it depends on which components carry the guarantee.
A few components are genuinely different in kind:
the runtime trace — the tool ran or it did not
an authoritative database — the record says X or it does not
a compiler, a test suite — the code builds or it does not
a schema / type check — the structure is valid or it is not
the deterministic policy engine — the rule fired or it did not
the enforcement point — the side effect executed or it did not
a scoped human verification — a person checked this claim, on the record
These do not share the generator’s failure modes, because they are not doing language understanding. When a guarantee rests on one of them, it is a real guarantee, bounded by its own assumptions (instrumentation is complete, the database is current, the policy is correct).
Everything else — the extractors, the judges, the geometric detectors, the critics — reduces correlated risk without eliminating it. Stacking several imperfect model-based checks helps to the extent their errors are not perfectly aligned. Their blind spots do not completely coincide (Chapter 8 §21), so a claim that slips past containment may still be caught by entailment or by provenance. But the residual is a joint failure of correlated components, and it does not shrink to zero by adding more of the same kind.
This is why the boundaries in this chapter are drawn where they are. The assertion, action, and persistence gates, and the enforcement point, are deterministic and sit outside the model on purpose. The stochastic components feed them evidence; they do not hold the authority. The thesis is not “the surrounding components are reliable.” It is narrower and it survives this objection:
Put the guarantee on the components that do not share the generator’s failure modes. Use the rest to lower the odds, and measure how much.
7. Contain stochasticity before consequential commitment
Not every component should be deterministic.
Generation, ranking, semantic retrieval, and some measurements gain their usefulness precisely from statistical generalization.
The design goal is not:
remove stochasticity.
It is:
Contain stochasticity inside components whose outputs cannot directly create consequential commitment.
“Consequential” matters more than “irreversible.”
An email can sometimes be recalled.
A database write can sometimes be rolled back.
A memory can sometimes be revoked.
They are still consequential transitions and deserve mediation.
8. Every consequential boundary should consume typed artifacts
A system becomes fragile when semantic distinctions live only in naming conventions and prose.
Prefer structured artifacts.
A candidate:
candidate = {
"candidate_id": "cand_v7",
"parent_id": "cand_v6",
"claims": ["c1", "c2", "c3"],
"evidence_snapshot": "evidence://bundle/1842",
"generator_version": "generator-v17",
}
A reliability record:
reliability = {
"containment": {...},
"structural_fidelity": {...},
"sensitivity": {...},
"epistemic_adequacy": {...},
"provenance": {...},
}
A policy decision:
policy_decision = {
"commitment": "HOLD",
"next_action": "VERIFY",
"obligations": ["VERIFY_PROVENANCE"],
"matched_rule": "PROVENANCE_REQUIRED",
}
A human decision:
human_decision = {
"decision": "APPROVE_ACTION",
"scope": "SEND_BOARD_UPDATE",
"reviewer_role": "finance_controller",
"evidence_seen": ["evidence_v2"],
}
A memory record:
origin
verification
valid time
system time
lifecycle
claim-level lineage
allowed uses
This prevents accidental semantics such as:
null means safe
absence means pass
retrieved means trusted
stored means true
PERMIT means persist
The architecture preserves distinctions in its types.
9. Evidence enters before authority
The evidence layer still begins with resolved objects:
claim
proposition
source
evidence item
support relation
provenance
freshness
conflict
A useful path is:
SOURCE / TOOL / USER INPUT
↓
NORMALIZE
↓
RESOLVE IDENTITY + SCOPE + TIME
↓
ATTACH PROVENANCE
↓
EVIDENCE SNAPSHOT
↓
OBSERVATION / MEASUREMENT
If the decision matters, preserve enough state to answer later:
Which source version?
Which support span?
Which retrieval result?
Which resolver version?
Which timestamp?
The rest of the architecture cannot recover distinctions discarded here.
10. Measurements remain sensors, not permissions
The book accumulated several possible measurement families:
containment
structural fidelity
consistency
sensitivity
answerability / epistemic adequacy
provenance
runtime/tool checks
No application needs all of them for every request.
But every deployed sensor needs a contract:
What property is targeted?
What observable is measured?
What assumptions connect them?
What are the blind spots?
What does PASS mean?
A containment signal cannot average away:
provenance = FAIL
A fluent answer cannot average away:
epistemic_adequacy = INSUFFICIENT_EVIDENCE
A context-sensitive answer cannot average away:
policy_restriction = TRUE
Never allow an average to cancel a non-negotiable failure.
11. Policy selects among feasible transitions
Policy does two things.
First, hard requirements eliminate impermissible transitions.
Conceptually:
Then the system can optimize among feasible actions for:
latency
cost
coverage
human burden
user interruption
So:
hard constraints
↓
feasible transitions
↓
optimize utility within feasible set
not:
average every risk and utility term together.
The policy function remains stateful:
first evidence gap
→ RETRIEVE
same gap after unsuccessful retrieval
→ REVIEW
budget exhausted
→ ABSTAIN
Runtime-governance work increasingly treats partial execution paths as part of the policy input rather than governing only the latest prompt.[3]
12. Enforcement must sit outside the model
A system prompt can say:
Do not send emails without approval.
That is steering.
It is not enforcement.
The stronger pattern is:
MODEL
proposes tool call
↓
POLICY DECISION
↓
POLICY ENFORCEMENT POINT
↓
execute / block / escalate
A minimal enforcement function is conceptually:
def enforce(tool_call, decision, approvals):
if decision.authorization != "PERMIT":
raise AuthorizationError("tool execution blocked")
if decision.requires_human_approval:
if tool_call.capability not in approvals:
raise AuthorizationError("human approval missing")
return execute(tool_call)
The executor consumes typed authorization.
It does not ask the model whether its own explanation sounds acceptable.
Recent tool-agent work demonstrates this general architecture by placing policy enforcement at the tool-call boundary rather than relying solely on prompts.[4]
Governance-by-construction work similarly inserts controls at multiple agent checkpoints.[5]
Prompt instructions shape proposals. Enforcement controls side effects.
13. Human review produces lineage, not magic truth
Humans are another governed resource.
They have:
latency
cost
fatigue
limited context
inconsistent judgment
capacity constraints
A reviewer should receive:
candidate
evidence
failed measurements
policy reason
repair history
lineage
proposed action
and return a typed decision.
An approval event can authorize an action within a declared scope.
It does not automatically establish every factual proposition present in the artifact.
If a reviewer performs explicit factual verification, record that as a separate verification artifact with:
reviewer role
claim scope
evidence seen
verification decision
time
policy/version context
That allows human intervention to participate in replayable lineage without turning humans into invisible stateless oracles.
14. Recovery is ordinary control flow
HOLD is not an exception.
It is a state.
HOLD
↓
LOCALIZE FAILURE
↓
RETRIEVE / VERIFY / REFINE / ASK
↓
NEW CANDIDATE OR EVIDENCE STATE
↓
INVALIDATE DEPENDENCIES
↓
RE-MEASURE
↓
POLICY
Recovery can terminate through:
PERMIT
DENY
ABSTAIN
REVIEW
NO_VALID_RECOVERY
budget exhaustion
cycle detection
Every repair creates a new candidate state.
Every evidence update creates a new evidence state.
Repeated attempts are not evidence that the next attempt is correct.
15. Persistence is its own state-changing subsystem
Memory candidates can originate from many places:
user assertions
tool observations
external sources
model candidates
agent trajectories
summaries
failure artifacts
human-reviewed state
They do not originate only from user-facing responses.
The write path is:
graph TD
US[USER / TOOL / SOURCE / MODEL / EXPERIENCE / FAILURE] --> MC[MEMORY CANDIDATE]
MC --> OVT[ORIGIN + VERIFICATION + TIME + LINEAGE + CAPABILITIES]
OVT --> MA[MEMORY ADMISSION POLICY]
MA --> PERS[PERSISTED WITH USE RIGHTS]
The write path treats persistence as a governed capability grant, not as a side effect of producing or observing text.
The read path is:
graph TD
Q[QUERY + PURPOSE + CALLER] --> AE[ACCESS ELIGIBILITY]
AE --> RET[RETRIEVAL]
RET --> EA[EVIDENTIAL ADMISSIBILITY]
EA --> UC[USABLE CONTEXT]
And the result loops back into proposal and measurement.
Chapter 14’s rule survives unchanged:
16. The ledger supports several different kinds of replay
The word replay hides different operations.
Exact replay
Run an immutable historical artifact through a deterministic component using the same inputs and version.
Example:
measurement record
+ policy-v2
→ deterministic policy decision
Counterfactual re-evaluation
Freeze the historical candidate and evidence, but intentionally change one component:
same candidate + evidence
policy-v1 → policy-v2
or:
same candidate + evidence
detector-v4 → detector-v5
This isolates the effect of the changed component.
Full-stack re-execution
Run the original request and captured external observations through the current system.
This may not exactly reproduce the old trajectory if:
generation is stochastic
hosted model implementation changed
seed control is unavailable
external tool behavior cannot be reconstructed
So the ledger should be honest about capability:
EXACT_REPLAY_AVAILABLE = false
COUNTERFACTUAL_REEVALUATION_AVAILABLE = true
FULL_STACK_REEXECUTION_AVAILABLE = true
For meaningful analysis, preserve where possible:
request
candidate versions
model IDs/configuration
random seed where meaningful
tool requests
tool responses
retrieval results
source versions
evidence snapshots
measurement versions
policy versions
human decisions
memory writes
side effects
And replay/re-execution must be dry-run by default:
no real email
no real payment
no memory write
no external mutation
The same enforcement logic should run, but side effects become recorded simulations.
17. Sensor disagreement and missing measurements are useful state
Suppose:
containment = PASS
structural_fidelity = PASS
sensitivity = FAIL
answerability = ANSWERABLE
provenance = VERIFIED
The wrong move is:
average → 0.81 → ACCEPT
The disagreement is the diagnosis:
factual support appears adequate
but the recommendation ignores decisive context.
Conversely:
containment = FAIL
support verifier = PASS
provenance = VERIFIED
may expose a sensor limitation.
Likewise, missing checks remain explicit:
UNAVAILABLE
NOT_MEASURED
NOT_RUN
UNCERTAIN
If progressive evaluation terminates early, record why:
state = NOT_RUN
reason = EARLY_TERMINATION_POLICY_BLOCK
not merely:
missing.
The application policy decides what these states imply.
The generator does not.
18. Evaluate paths and attribute the first failed boundary
After fifteen chapters, end-to-end correctness cannot mean only:
Did the final sentence look right?
A system can fail in:
retrieval
resolution
measurement
verification
policy
enforcement
repair
memory
human review
A behavior specification can say:
GIVEN
Q3 revenue is absent from admissible evidence
WHEN
the model proposes a numerical Q3 value
THEN
assertion commitment = HOLD
next action = RETRIEVE
memory admission = QUARANTINE
external send = BLOCKED
A paired case adds authoritative Q3 evidence:
THEN
supported assertion may be PERMITTED
memory may be separately ADMITTED
external send still follows action policy
System-level failures should also record:
first_invalid_state
first_failed_boundary
propagated_failures
final_consequence
For example:
FIRST FAILURE
retrieval returned stale evidence
DOWNSTREAM
support verifier correctly evaluated the stale snapshot
policy correctly acted on its input state
CONSEQUENCE
incorrect assertion emitted
That separates root cause from downstream propagation.
Bhagat et al.’s compound-system framework similarly argues for explicit behavior specifications and scenario tests rather than aggregate benchmark scores alone.[1]
19. Test the boundaries directly
The most important tests are often stronger than language-quality tests.
Assertion
unestablished claim
→ cannot be emitted as verified fact
Action
prohibited tool proposal
→ cannot execute even if model insists
Enforcement
policy = DENY
executor receives tool call anyway
→ side effect still impossible
Persistence
HOLD candidate
→ cannot become active factual memory
Lineage
unverified ancestor
→ summarization cannot erase obligation
Recovery
repair
→ no inherited authorization
Unknown state
NOT_MEASURED
→ never silently treated as PASS
Budget
retry limit reached
→ deterministic terminal route
These are boundary invariants.
They are often easier to establish strongly than the model’s internal reasoning.
20. Reliability is a consequence tax, not a blanket setting
A full Level-7 reliability path is wasteful for:
Write a limerick about a dog.
The same shortcut can be reckless for:
Send a financial update to the board.
So the control plane should scale with consequence.
A simple risk router can consider:
factual consequence
action blast radius
persistence duration
reversibility
privacy sensitivity
financial/legal exposure
human impact
Then choose a path.
Fast path
Suitable for low-consequence tasks:
basic policy restrictions
cheap deterministic checks
generation
Governed path
For higher consequence:
source/provenance checks
typed measurements
external verification
strong action mediation
memory admission
human review where required
This is the consequence tax:
The more expensive a failure is, the more reliability work the system should be willing to buy before commitment.
This does not mean every expensive check belongs at the end.
Progressive evaluation should order work using both:
blocker severity
and
expected information gained per unit cost/latency
Hard cheap blockers go early.
Expensive sensitivity suites and human review can be reserved for cases where they can still change the decision.
Measure the resulting path against:
P50/P95/P99 latency
API cost
human-review capacity
coverage
false acceptance
false rejection
21. Monitor the control plane, not only the model
Operational observability should track the distribution of decisions.
Suppose historically:
RETRIEVE = 2%
REVIEW = 0.5%
ABSTAIN = 1%
and suddenly:
RETRIEVE = 45%.
That is not automatically evidence that the system became safer.
Possible causes include:
vector database outage
embedding model change
knowledge-base deletion
provenance resolver bug
policy change
source staleness spike
Monitor distributions such as:
PERMIT / HOLD / DENY
RETRIEVE / VERIFY / ASK / REVIEW
human approval rate
memory ADMIT / QUARANTINE
NOT_RUN / UNAVAILABLE
matched-rule frequency
first-failed-boundary frequency
Alert on meaningful shifts.
The control plane can drift operationally even when no model version changes.
22. Practical adoption is incremental—but not monotonically safer
A minimum useful architecture is:
1. version consequential candidates
2. preserve evidence and provenance
3. use explicit typed state at every consequential boundary
4. separate policy from generation
5. mediate consequential tools
6. gate persistent memory
7. retain enough lineage for replay
8. turn incidents into regression tests
A maturity ladder can then guide implementation.
Level 0 — Raw generation
prompt → model → output
Level 1 — Grounded generation
retrieval → model
Exit test:
Can we identify exactly which evidence entered the request?
Level 2 — Measured generation
candidate → typed sensors
Exit test:
Can every required measurement report explicit state,
including NOT_MEASURED / UNAVAILABLE?
Level 3 — Policy-mediated output
reliability record → policy
Exit test:
Can we replay a policy version and explain the decision diff?
Level 4 — Governed actions
tool proposal → external enforcement
Exit test:
Can we prove a prohibited tool call cannot execute
even when the model proposes it?
Level 5 — Bounded recovery
verify → repair → re-measure → re-authorize
Exit test:
Does every recovery loop terminate under budget/cycle rules?
Level 6 — Governed memory
write/read admission + lineage + revocation
Exit test:
Can a HOLD/model-derived candidate be prevented
from becoming active factual memory?
Level 7 — Replayable reliability engineering
behavior tests
policy replay
measurement replay
incident regression
control-plane monitoring
This is a capability/governance ladder, not a theorem that each added subsystem is automatically safer.
Level 0 → 1 is the clearest example. Adding retrieval gives the model grounding, but it also opens an attack surface that raw generation did not have: a document in the corpus can now carry an injected instruction or a planted false fact straight into the context. A Level-1 system with an unfiltered retriever can be less safe on adversarial inputs than the Level-0 system it replaced. The same holds at Level 5 → 6 — persistence is the channel Chapter 14 is entirely about.
So the ladder measures capability, not safety. Each rung adds a subsystem, and each subsystem has failure modes that must be evaluated in its own right. Do not read a level number as a safety rating.
23. The complete architecture is a closed governed system
The main control loop is:
graph TD
R[REQUEST] --> CR[CONTEXT + RISK PROFILE]
CR --> ER[EVIDENCE / MEMORY READ PATH]
CR --> PP[PROPOSAL PLANE]
ER --> OR[OBSERVATION RECORD]
PP --> VC[VERSIONED CANDIDATE]
OR --> OM[OBSERVATION / MEASUREMENT]
VC --> OM
OM --> RR[RELIABILITY RECORD]
RR --> CP[CONTROL PLANE]
CP --> HOLD[HOLD]
CP --> PER[PERMIT]
CP --> DEN[DENY]
PER --> AG[ASSERTION GATE]
PER --> ACT[ACTION GATE]
AG --> UA[USER/API]
ACT --> EN[ENFORCEMENT]
EN --> TR[TOOL RESULT]
TR --> NO[NEW OBSERVATION]
NO --> OM
HOLD --> RL[RECOVERY LOOP]
RL --> OM
DEN --> AB[ABSTAIN / REJECT]
The closed loop keeps requests, evidence, candidates, measurements, policy decisions, enforcement, and recovery in one governed runtime process.
Persistence is a separate state-changing loop:
graph TD
US[USER / TOOL / SOURCE / MODEL / EXPERIENCE / FAILURE] --> MC[MEMORY CANDIDATE]
MC --> PPOL[PERSISTENCE POLICY]
PPOL --> PM[PERSISTED WITH USE CAPABILITIES]
PM --> FS[FUTURE STATE]
FS --> FRP[future read path]
Running alongside both loops is the reliability ledger:
EVIDENCE SNAPSHOTS
CANDIDATE LINEAGE
MEASUREMENT RECORDS
POLICY DECISIONS
HUMAN DECISIONS
ACTION EXECUTION
MEMORY EVENTS
VERSION HISTORY
The architecture is therefore not a terminal pipeline.
Actions create observations.
Memory creates future input.
Repair creates new candidates.
Policy changes future paths.
The whole system is a governed feedback process.
A useful compression is:
24. AI can help design the system without owning the floor
AI is useful throughout this architecture:
candidate generation
claim decomposition
memory proposal extraction
repair proposals
retrieval query generation
counterfactual tests
failure clustering
policy-test generation
human-review summarization
But a dangerous shortcut remains:
Give the reliability JSON to a cheap LLM
and ask whether to ACCEPT.
That puts the policy floor back inside the probabilistic component.
A similarly dangerous generated policy is:
if record["state"] != "FAIL":
allow()
which silently turns:
NOT_MEASURED
UNAVAILABLE
UNCERTAIN
into permission.
The model can propose rules.
The rules should then be translated into explicit executable semantics and tested.
Where requirements can be expressed mechanically:
verified provenance required
human approval required
retrieval attempts <= 2
FACTUAL_EVIDENCE capability forbidden
keep the final enforcement floor outside unconstrained model judgment.
25. Ten laws—and where they came from
Law 1 — Generation is not acceptance
Generation may be stochastic. Acceptance does not have to be.
From Chapters 1 and 12.
Law 2 — A sensor is not a verdict
measurement
≠
authorization
From Chapters 4 and 6.
Law 3 — Preserve distinctions before scoring
If a decision needs:
polarity
role
quantity
time
provenance
preserve them until after the decision.
From Chapter 8.
Law 4 — Different failures need different recovery
missing user input → ASK
missing evidence → RETRIEVE
conflict → VERIFY
bad candidate → REFINE
unrecoverable → ABSTAIN / REJECT
From Chapters 11–13.
Law 5 — Every repair creates a new state
repair
→ re-measure
→ re-authorize
From Chapter 13.
Law 6 — Relevance is not admissibility
A retrieved item can be highly relevant and still be forbidden as evidence.
From Chapter 14.
Law 7 — Transformation does not manufacture evidence
Summarization, rephrasing, repetition, storage, and retrieval do not by themselves create independent factual support.
From Chapters 13–14.
Law 8 — Side effects need external mediation
The model proposes actions.
A trusted runtime decides whether they execute.
From Chapters 12 and 15.
Law 9 — Persistent state needs its own gate
response PERMIT
≠
memory ADMIT
From Chapter 14.
Law 10 — Reliability claims must be scoped
Never reduce the conclusion to:
this model is trustworthy.
State:
workload
policy version
measurements
evidence assumptions
enforcement boundary
observed failure regime
From Chapters 6, 12, and 15.
26. What the book removed
The book’s progression can be read as a sequence of hidden assumptions that no longer survive.
| Chapter | Assumption removed |
|---|---|
| 1 | Fluency is verification |
| 2 | Hallucination is one failure class |
| 3 | Evidence, truth, support, provenance and verification are interchangeable |
| 4 | A metric is a verdict |
| 5 | Semantic containment is proof of truth |
| 6 | Detector quality equals deployment utility |
| 7 | Average benchmark performance is enough |
| 8 | Lost structure can be recovered from a scalar |
| 9 | Reliability means invariance to every change |
| 10 | Safe-looking generic output is useful reasoning |
| 11 | Always answering is a capability |
| 12 | Measurements can authorize themselves |
| 13 | A repair inherits trust |
| 14 | Stored or repeated text becomes evidence |
| 15 | The model is the system |
The architecture is what remains once those shortcuts are removed.
27. What you should now be able to answer
Why is assertion PERMIT not enough to send an email?
Because assertion and action are different commitment boundaries. The first authorizes a content claim to be emitted. The second authorizes a side effect. An externally consequential action can still require human approval, capability checks, or other obligations after the response itself is factually authorized.
Why is response PERMIT not enough to store factual memory?
Because memory is future state. Persistence requires its own admission policy over origin, verification, time, lineage, and future-use capabilities. A permitted response is only eligible to become a memory candidate.
Why can deterministic policy still produce a bad outcome?
Because policy is deterministic given its inputs. If risk classification, entity resolution, verification, or another policy input is wrong, deterministic arbitration can consistently authorize the wrong transition. Reliability claims must therefore state both policy semantics and input assumptions.
Why is a tool result not simply the end of an action?
Because tool execution changes or observes the world. The result becomes a new observation that can alter evidence, invalidate measurements, change the candidate, and trigger another policy decision. The architecture is a feedback system, not a one-way pipeline.
Why is control-plane distribution monitoring a reliability concern?
Because sudden changes in HOLD, RETRIEVE, REVIEW, or QUARANTINE rates can reveal failures in retrieval, evidence stores, policy configuration, or verification even when the model has not changed. Policy decisions are operational signals.
What is the strongest reliability claim this architecture supports?
A bounded claim such as:
Under policy P,
validated inputs I,
evidence assumptions E,
and enforcement implementation F,
action A cannot cross boundary B
unless rule R is satisfied.
It does not support the global statement:
the AI is safe.
28. Deployment questions
Before deploying a consequential workflow, the boundary tests in Section 19 must pass: a prohibited tool call cannot execute, a HOLD candidate cannot become factual memory, an unverified ancestor’s obligation survives summarization, NOT_MEASURED is never read as PASS, and every recovery loop terminates.
Then ask the questions the boundary tests do not cover:
Evidence Can we reconstruct the evidence that existed at decision time?
Candidate Are consequential outputs versioned rather than invisibly mutated?
Policy Which constraints are hard blockers, and which transitions remain feasible?
Human review Is approval scoped, typed, and recorded with the evidence the reviewer saw?
Replay Which decisions are exactly replayable, counterfactually re-evaluable,
or only re-executable?
Operations Are policy-decision and route distributions monitored for drift?
Evaluation Do behavior tests exercise complete paths and record the first failed boundary?
If these have no explicit answers, the application is still depending on invisible trust.
29. Exercises
Exercise 1 — Build the three boundaries
Create a factual assistant that can also send email and store memory.
Implement separate:
assertion_authorization
action_authorization
memory_admission
Construct one case in which all three return different outcomes.
Exercise 2 — Test the enforcement point
Make the model propose:
SEND_EMAIL
while policy returns:
DENY
Attempt to bypass the policy layer and send the call directly to the executor.
The enforcement point should still prevent the side effect.
Exercise 3 — Whole-system counterfactual
Create:
A: decisive evidence absent
B: decisive evidence present
Expected behavior:
A → HOLD + RETRIEVE / ABSTAIN
B → assertion may PERMIT
Evaluate assertion, action, and persistence separately.
Exercise 4 — Replay without side effects
Store:
candidate
evidence snapshot
measurement record
policy-v1 decision
Create policy-v2 with one stricter provenance requirement.
Re-evaluate historical cases in dry-run mode.
Confirm that no external tool or memory write executes during replay.
Exercise 5 — Attribute the first failed boundary
Construct an end-to-end failure caused by stale retrieval.
Record:
first_invalid_state
first_failed_boundary
propagated_failures
final_consequence
Verify that a later policy decision is not incorrectly blamed for a failure that originated upstream.
Exercise 6 — Monitor policy drift
Generate a baseline distribution over:
PERMIT
HOLD
RETRIEVE
VERIFY
REVIEW
QUARANTINE
Then simulate a retrieval outage.
Measure which routing distributions move and design an alert.
30. The deeper lesson
Return to the fictional Northbridge Medal from Chapter 1.
The model that could fluently invent an answer there can sit inside this final architecture unchanged.
It may still invent.
The difference is that the invention no longer receives invisible authority merely because it was fluent.
It can be:
measured
held
verified
rejected
repaired
prevented from executing
prevented from persisting
without requiring the generator itself to become infallible.
That is the systems move.
A language model is extraordinarily useful precisely because it can generate beyond what has been explicitly programmed.
The same property means it should not be treated as unquestioned authority over everything it generates.
Build systems in which:
a model can be wrong
without the database becoming wrong;
a model can propose a bad action
without the action executing;
a model can generate an unsupported claim
without the claim becoming future evidence;
a verifier can be uncertain
without uncertainty silently becoming permission;
a repair can fail
without the repaired answer inheriting trust;
a policy can change
without losing the ability to re-evaluate old decisions.
The final loop is:
MODEL PROPOSES
↓
SYSTEM OBSERVES
↓
SYSTEM MEASURES
↓
SYSTEM DECIDES
↓
SYSTEM ENFORCES
↓
SYSTEM RECORDS
↓
INCIDENTS BECOME TESTS
The model remains at the center of the application.
It is no longer the center of authority.
The reliable unit is not the model. It is the system around the model.
Its reliability no longer depends on pretending the model is infallible.
It can instead earn bounded reliability under explicit assumptions while the model remains fallible.
Research roots
-
Pranav Bhagat, K N Ajay Shastry, Pranoy Panda and Chaitanya Devaguptapu, “Evaluating Compound AI Systems through Behaviors, Not Benchmarks,” Findings of EMNLP 2025, pp. 24193–24222. Proposes behavior-driven evaluation specifications for compound AI systems and reports that scenario-oriented tests expose failures missed by conventional benchmark evaluation. https://aclanthology.org/2025.findings-emnlp.1314/
-
C. Brian Smith and Daniel McCarthy, “Deterministic governance for generative systems: a policy-aligned runtime for validating AI outputs,” AI and Ethics 6, article 394, 2026. Describes a bounded deterministic arbitration architecture with versioned constraints and evidence records for replayable governance decisions. https://doi.org/10.1007/s43681-026-01172-6
-
Maurits Kaptein, Vassilis-Javed Khan and Andriy Podstavnychy, “Runtime Governance for AI Agents: Policies on Paths,” arXiv:2603.16586, 2026. Formalizes runtime governance over partial execution paths and proposed next actions, emphasizing that path-dependent behavior cannot be governed solely by static prompts or static access control. https://arxiv.org/abs/2603.16586
-
Shanshan Wang, Sizheng Zhu and Rende Li, “Runtime Policy Enforcement for MCP-Based LLM Agents,” Electronics 15(13), 2829, 2026. Implements a policy enforcement point at the tool-call boundary with declarative rules, cross-step information-flow labels, and audit logging. https://doi.org/10.3390/electronics15132829
-
Segev Shlomov, Iftach Shoham, Alon Oved, Ido Levy, Sami Marreed, Harold J. Ship, Offer Akrabi, Sergey Zeltyn, Avi Yaeli and Nir Mashkif, “Governance by Construction for Generalist Agents,” Proceedings of the ACM Conference on AI and Agentic Systems, 2026, pp. 1280–1287. Demonstrates policy-as-code interventions across multiple agent execution checkpoints, including intent, tool use, human approval and output. https://doi.org/10.1145/3786335.3813192
-
Chloe Autio, Reva Schwartz, Jesse Dunietz, Shomik Jain, Martin Stanley, Elham Tabassi, Patrick Hall and Kamie Roberts, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1, 2024, updated 2026. Frames generative-AI trustworthiness as lifecycle risk management involving governance, measurement, management and ongoing evaluation. https://doi.org/10.6028/NIST.AI.600-1
End
The book began with hallucination as a model behavior.
It ends with reliability as a systems property.
That is the shift.
