Part 3 — Give Intelligence a Runtime

Three sentences and a merge

A review of a cache change says three things:

The cache TTL is set to 60 seconds in config/cache.toml. The retry test passes.
p99 latency stays under 200 ms.

Someone reads it and merges the change. The next day, the deployed configuration turns out to say 600 seconds.

The obvious question is whether the review was wrong. The more useful question is narrower: which part of the decision rested on which sentence, and what had anyone actually checked? One sentence could be confirmed against a file. One could be tested. One had no evidence at all. The merge used two of them and knowingly left the third open, or it didn’t. The merge record lists no basis, so the basis cannot be reconstructed.

Before this chapter, CodeAI could not say either. It had the vocabulary: evidence classes from E0 (asserted) to E4 (robust), claim statuses, even a Decision type. A probe at the Stage 17 commit showed what the vocabulary was worth:

• Asserted evidence was accepted. A claim recorded with evidence class E4_ROBUST and status “supported”, citing a source artifact that did not exist, was stored and projected exactly as asserted.
• Any passing command promoted. A claim was promoted to E3_REPRODUCED by a check whose command printed “nothing tested”.
• No decisions. There was no way to record one at all.

Reading the old projection’s source adds two more, not part of the preserved run. A later claim.recorded for the same ID replaces the claim, refuted or not. And promotion events are accepted from any actor.

Which statements are supported, by what, under which conditions?

Said, supported, relied on

Three things are easy to collapse into one word, “claim”.

Said. Someone, or some model, stated something, at a place you can point to. The W3C provenance model gives this a precise name. In its words, “Attribution is the ascribing of an entity to an agent” (Moreau and Missier, 2013). It separately defines a quotation as the repeat of some or all of an entity. Attribution tells you who said it and where. It tells you nothing about whether it is true.

Supported. Something other than the statement bears on it. FEVER, a large fact-verification dataset, labels each claim Supported, Refuted or NotEnoughInfo (Thorne et al., 2018). For the first two labels, annotators also recorded the sentences that formed the evidence. The third outcome matters as much as the other two: a claim with no evidence is not false, it is unresolved. FActScore breaks long generations into atomic facts and measures the share supported by a knowledge source (Min et al., 2023). Its central observation is that one fluent answer can hold supported and unsupported statements side by side.

Relied on. A decision used the statement. This is the part the evaluation literature doesn’t cover, because benchmarks score claims and don’t act on them. A system that acts has a further duty: to record which claims it used, in what state, so that a later change to those claims can be traced to the decision.

None of these sources is about model runtimes, and CodeAI does not do what FEVER and FActScore do. Their verdicts come from annotators or from a model estimating support. CodeAI makes no judgment of entailment at all. It checks that evidence exists where it says it does, and who recorded it. The judgment that a passage bears on a claim stays with a named actor. The mapping in this chapter is the book’s.

Each rung is a separate record; standing is derived, never stored — and the top rung has no path that builds it:

    flowchart LR
    E0["E0 · asserted<br/><i>claim.recorded — starts unresolved</i>"] --> E1["E1 · attributed<br/><i>extract: exact span of preserved bytes</i>"]
    E1 --> E2["E2 · source checked<br/><i>passage in a different source,<br/>or a completed check naming the claim</i>"]
    E2 --> E3["E3 · reproduced<br/><i>independent check passed</i>"]
    E3 -.->|"unreachable: no path builds it"| E4["E4 · robust"]
    E3 --> ST["standing<br/><i>derived, never stored</i>"]
    ST --> DC["decision<br/><i>snapshots the standing relied on;<br/>reports intact or changed, never edits</i>"]
    style E4 stroke-dasharray: 4 4
  

What CodeAI now does

CodeAI 3b6d8fb adds src/codeai/evidence.py and five runtime operations beside the old claim API, which is left as it was. The new projections ignore the old API’s events and list them, so a reader sees both.

The shape is clearest in the calls the experiment actually made. Reduced from the executed producer, with comments naming the positional arguments:

# Said: an exact span of a preserved response. No evidence class, no status.
runtime.extract_claim(ClaimExtraction(
    "c-ttl", TASK, call_id, attempt_id,
    0, 56, quote,                          # span start, span end, quoted text
    statement, "claim-extractor"))

# Supported: a passage of a different preserved source, cited by a named actor...
runtime.record_claim_evidence(EvidenceRecord(
    "ev-config", "c-ttl", SOURCE_PASSAGE, SUPPORTS, "human-reviewer",
    source_artifact_id=config.artifact_id,
    passage_start=start, passage_end=start + len("ttl_seconds = 60"),
    passage="ttl_seconds = 60"))

# ...or a completed check whose request named the claim.
runtime.record_claim_evidence(EvidenceRecord(
    "ev-retry", "c-retry", CHECK, SUPPORTS, "verification-reviewer",
    check_id="check-retry"))

# Relied on: the claims a decision uses, and the ones it knowingly leaves open.
runtime.record_decision(DecisionRequest(
    "dec-merge", TASK, "release-manager", "Merge the cache change",
    ("c-ttl", "c-retry"),                  # relied on
    ("c-latency",)))                       # acknowledged open

runtime.decision_standing("dec-merge")     # basis_intact or basis_changed, naming what moved
runtime.decisions_resting_on("c-ttl")      # every decision that used this claim

extract_claim reads the response bytes the runtime preserved — the same bytes Chapter 17 insists must outlive every interpretation — checks their hash, and refuses unless the quote equals the output text at that span. The claim is causally linked to the observation and starts unresolved at E1_ATTRIBUTED, whatever it says. The caller cannot supply an evidence class or a status.

record_claim_evidence accepts two kinds of evidence. A source passage (E2_SOURCE_CHECKED) must be an exact passage of a preserved, hash-matched artifact that is not the claim’s own response: a review cannot cite itself. A check (E3_REPRODUCED) must be a completed check whose request named the claim; PASS may support, FAIL may refute, and anything else is inconclusive. Either way, the actor that produced the claim’s source call may not record evidence for it.

Standing is derived, never stored. claim_standing reads status from the evidence — supports only is supported, refutes only is refuted, both is contested, neither is unresolved — takes the strongest support as the evidence class, and adds the source call’s adopted status, which Chapter 17 made revisable.

record_decision refuses unless every relied-on claim is supported, at E2 or better, from a source call whose adopted status is succeeded. Claims the decision leaves open are named separately and never relied on. The decision stores a snapshot of each relied-on claim’s standing, and decision_standing and decisions_resting_on compare those snapshots with current standing. They report basis_intact or basis_changed, name every field that moved, and never edit or revoke the decision.

Every refusal is appended as its own event — claim.refused, claim.evidence_refused, decision.refused — and raised. Nothing else is appended.

The experiment

The run was preregistered, then executed at CodeAI 3b6d8fb from a clean worktree. Every step — review, extract, evidence, decide, contradict, reinterpret, inspect — ran as its own operating-system process, so nothing survived between steps except what the ledger and artifact store held. A synthetic provider logged every request to a receipt log outside the ledger, outbound connections were refused, and the checks were real local commands: parsing the TOML file and a small retry test. The review itself is a synthetic fixture.

Claims that start unresolved

The review call produced one provider receipt and ten events, and a separate process then extracted three claims — characters 0–56 (the TTL), 57–79 (the retry test) and 80–111 (latency) — all unresolved at E1_ATTRIBUTED.

The same process tried to extract a fourth claim, quoting the review as saying “600 seconds”. The quote did not match the preserved text at that span, so the attempt was refused for quote_not_in_source, with only the refusal in the ledger and no claim created. Repeating a real extraction appended nothing, and with no further provider requests after the review, the receipt count stayed at one through every later step.

Evidence that counts, and evidence that doesn’t

Counted, from a source passage. A human reviewer cited the passage ttl_seconds = 60 in the preserved config/cache.toml. The TTL claim became supported at E2_SOURCE_CHECKED.

Three attempts were refused, each appending only its refusal:

• Self-evidence. The reviewing model recorded the same passage as evidence for its own claim: self_evidence.
• Circular evidence. The human reviewer cited the review’s own response bytes as the source for the TTL sentence. The passage was really there, but a statement does not become evidence for itself by being quoted: source_is_claim_origin.
• An untargeted check. A check that parsed the TOML file, naming only the TTL claim, passed. Cited as evidence for the retry claim, it was refused: check_not_targeting_claim. Before this stage, a passing check was enough.

Counted, from a check. The retry test ran (exit 0, “retry test passed”) under a check request that named the retry claim. Cited for that claim, it made the claim supported at E3_REPRODUCED.

run_check still appends its old promotion events as it always did. The new standing ignores them and lists their IDs, so the discrepancy is visible rather than silent.

A decision with its reasons attached

A release manager first tried to ship on the TTL and latency claims. Refused: claim_not_supported:c-latency and evidence_below_policy:c-latency. The refusal names the claim and records what the manager was trying to rely on.

The second decision, “Merge the cache change”, relied on the TTL claim (supported, E2) and the retry claim (supported, E3). It acknowledged the latency claim as unresolved. It was recorded with a snapshot of both relied-on claims:

• status and evidence class
• the evidence IDs for and against
• the quote and observation hashes
• the source call’s adopted status and the record it came from

Repeating it appended nothing.

That acknowledgement is the difference between “we didn’t check latency” and “we didn’t notice we hadn’t checked latency”. A decision that must list what it leaves open cannot quietly lean on it.

Day two: the source disagrees

A new process recorded one more piece of evidence. An on-call engineer cited ttl_seconds = 600 in the deployed configuration, a different preserved file, as refuting the TTL claim. The claim became contested. It still has its E2 support; it now has a refutation too, and CodeAI does not choose between them.

The decision’s standing became basis_changed, with exactly two changes on the TTL claim — status: supported → contested, and refuting_evidence_ids: none → ev-deployed — while the retry claim’s entry did not change.

Asked which decisions rest on the TTL claim, the runtime names the merge. The retry claim names it too without a change to its entry, and the latency claim names none, because the merge acknowledged it and never relied on it.

The decision.recorded event is byte-for-byte the event recorded the day before, and so are all 27 events before it. The decision is not rewritten to fit what is now known. It is shown to have been made on a basis that has since moved, and exactly where.

When the source call is reinterpreted

Chapter 17 showed that a call’s adopted status can change after the fact. This run joins the two chapters.

1. Day one. The review was recorded as succeeded under the old interpreter, although its finish reason was length.
2. Claims, evidence, decision. Claims were extracted, both were supported as before, and the merge decision was recorded.
3. Reinterpretation. reinterpret_call then read the preserved bytes under v2 and adopted unresolved, without a provider request.

The decision’s standing changed on source_call_status, succeeded → unresolved, for both claims, along with the status record each now rests on. The evidence had not changed. The source had.

An identical decision recorded afterwards was refused for source_call_not_succeeded on both claims.

Agreement is not evidence

Two different models, on two separate calls, returned the same TTL sentence. Both claims were attributed and both stayed unresolved at E1. A decision relying on “two models agree” was refused with four reasons: neither claim was supported, and neither met the evidence policy.

Two quotations of an unchecked statement are still two unchecked statements. Agreement can tell you what to check next. It is not the check.

Without the bytes

Two copies were taken right after the review: one with the response body deleted, one with its bytes altered. All five extraction attempts on each were refused for observation_unavailable, and no claim was created.

A claim that points at text nobody can read back is not attributed. It is only asserted.

Checking it without trusting it

The bundle’s verifier imports neither CodeAI nor the producer, and applies its own rules. It reads every preserved response and checks every quote at its span. It validates every evidence item against source bytes or check records. It re-derives every claim standing, decision basis, basis change, and refusal reason from the events exported at the same moment. And it requires every later export to extend the earlier one unchanged, with provider receipts coming only from review steps.

All 17 semantic claims pass. The full run, including byte hashes for every file in the bundle, exits 0. A fresh CodeAI process reopened copies of three ledgers and reproduced every recorded standing.

Five seeded corruptions were run with the byte inventory bypassed:

What this is not

• Fact-checking is out of scope. CodeAI confirms that a passage exists in a different, preserved source and who cited it. Whether the passage entails the claim is that person’s judgment, recorded under their name.
• Not claim extraction. Spans were supplied. Nothing here splits text into atomic facts, and the statement’s paraphrase is not checked, only the quote.
• Reporting, not enforcement. A changed basis is reported, without blocking action on it.
• Not truth maintenance. Evidence is never retracted, and a newer source does not outrank an older one. Two disagreeing sources simply make a claim contested.
• Not a fix of the old API. record_claim still stores whatever it is told, and experiments still use it. Decisions refuse to rely on its claims.

Where it is still weak

1. The self-evidence rule compares actor IDs. The same model or person under a different ID passes.
2. A check supports a claim because the claim was named in its request. Whether the command tests the claim is not examined.
3. The decision policy is fixed at E2, and E4_ROBUST is unreachable.
4. Spans are character offsets into text derived from the bytes by CodeAI’s output reader, not byte offsets into the raw response.
5. One synthetic review, local checks, a single writer.

Do this now

Thirty minutes. Take one decision made in the last week on the strength of an AI answer.

1. List the statements the decision actually used. For each one, point to where it was said: a message, a line, a file. If you can’t point to it, it is asserted, not attributed.
2. For each statement, write what supported it: a document passage you read, a test you ran, or nothing. Mark anything supported only by “the model said so” or “two tools agreed” as unresolved.
3. Mark the statements the decision knowingly left unresolved. Were any of them relied on anyway?
4. Suppose one supporting source changed tomorrow. Could you find this decision from that source, without remembering it?

If you are building with an assistant:

Separate what was said, what supports it, and what a decision relied on.
- Record a claim only as an exact quote at a span of preserved output,
  hash-checked; it starts unresolved. Refuse quotes not at the span.
- Record evidence as its own event: a passage of a different, preserved,
  hash-matched source, or a completed check whose request named the claim.
  Refuse evidence recorded by the claim's producer, evidence citing the
  claim's own output, and checks that did not target the claim.
- Derive status (supported/refuted/contested/unresolved) and evidence
  class from evidence records. Never store an asserted status.
- Record a decision with the claims it relies on and those it leaves
  open; refuse reliance on unsupported claims; snapshot their standing.
- Add a projection that compares each decision's snapshot with current
  standing and names what changed. Never edit the decision.
- Test in separate processes: refusals appended and nothing else; a later
  refutation changes the decision's standing and not its record.

Failure modes

• Treating attribution as support. It records who said it, not whether it holds.
• Letting agreement stand in for evidence. Two unchecked quotations are still unchecked.
• Letting any passing check promote. The check must be about the claim.
• Citing the output as its own evidence. A quote of the claim is not a source for it.
• Storing a status instead of deriving it. Whoever writes last decides what is true.
• Deciding without recording the basis. When a source changes, no one can find what rested on it.
• Rewriting the decision when the basis moves. Report the change; keep what was decided and why.

What this chapter established

• Said ≠ supported ≠ relied on: attribution (PROV-DM), support with an honest not-enough-information outcome assessed claim by claim (FEVER, FActScore), and reliance — which claims a decision used, in what state. The runtime mapping is the book’s.
• CodeAI 3b6d8fb attributes claims to exact spans of preserved responses, validates evidence as a different source’s passage or a check that named the claim, derives standing from evidence, and records decisions with their relied-on claims, their acknowledged open claims, and a basis snapshot.
• In separate processes, a forged quote, self-authored evidence, circular evidence, an untargeted check, model agreement, and missing bytes were all refused and recorded, and a decision relying on an unresolved claim was refused.
• A recorded decision’s standing changed when a deployed configuration refuted one claim, and separately when its source call was reinterpreted; the decision record stayed byte-for-byte unchanged and named exactly what moved.
• An independent verifier passed 17 semantic claims plus file hashes and rejected five seeded corruptions.
• Not established: entailment, automatic extraction, enforcement, source precedence, or a fix of the legacy claim API — which, before this stage, accepted a fabricated E4 claim and still exists.

Next

A decision can now say what it rested on. It still changes nothing. Merging, deploying and writing a file are effects, and Chapter 16 showed that effects cannot be taken back.

The next step is letting a decision request a change to the world, under explicit authority. That leaves an open question: what happens when the basis of a decision moves after the action has already run?

Continue with Let the Machine Touch Something.

References

• Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. EMNLP, 2023. arXiv:2305.14251. https://arxiv.org/abs/2305.14251
• Luc Moreau and Paolo Missier (eds.). PROV-DM: The PROV Data Model. W3C Recommendation, 30 April 2013. https://www.w3.org/TR/prov-dm/
• James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: a large-scale dataset for Fact Extraction and VERification. NAACL-HLT, 2018. arXiv:1803.05355. https://arxiv.org/abs/1803.05355

Implementation sources:

• CodeAI 3b6d8fb:
  • src/codeai/evidence.py: ClaimExtraction, EvidenceRecord, DecisionRequest, ClaimStanding, DecisionStanding, extract_claim, record_evidence, project_claim_standing, record_decision, project_decision_standing, decisions_resting_on, CLAIM_EVIDENCE_V1, DECISION_BASIS_V1.
  • src/codeai/runtime.py: extract_claim, record_claim_evidence, claim_standing, record_decision, decision_standing, decisions_resting_on.
  • Tests: tests/test_claim_evidence.py (14); full suite 341 passed.
• Evidence: experiments/applied-ai/evidence/claims-evidence/2026-09-14-3b6d8fb/.
  • Preregistration and execution record.
  • Six cases, each with an SQLite ledger, artifact store, provider receipt log, workspace, per-step action records, inspections and exported events.
  • The independent verify.py, five seeded corruptions, test outputs, chapter-evidence-report.md and hashes.json.
• Producer: experiments/applied-ai/claims_evidence_demo.py. The executed copy is pinned as the bundle’s run.py.