Part 3 — Give Intelligence a Runtime

The success that was cut off

Chapter 11’s fourth live run was recorded as a success. The provider had answered with HTTP 200 and text, and CodeAI’s interpreter at the time treated any answer with text as complete. Opening the preserved response told a different story: "finish_reason": "length". The model had used every output token it was allowed, and the review stopped mid-sentence.

That mistake was found by a person, reading bytes that happened to have been kept. Chapter 12 then taught the adapters to read the finish reason. Chapter 13 built better readings as pure projections over preserved responses, and stopped deliberately short of letting them change anything: past decisions stayed bound to the interpretation in force when they were made.

So the record still said “succeeded”. A better interpreter existed, and nothing in the runtime could act on it.

Why must the underlying observation outlive the interpretation?

Observe today, interpret tomorrow

The honest sequence in a system that calls models looks like this. Observe today. Interpret today. Discover tomorrow that the interpreter was wrong. Then interpret again, tomorrow, without repeating yesterday’s request.

Yesterday’s request is not repeatable in any useful sense. It was paid for, and it was stochastic. Asking again produces a different generation, possibly a complete one, and the question “what did we actually receive on the day we decided?” can no longer be answered. It went out over a network, and Chapter 16 argued that a served provider request cannot be taken back.

Chapter 16 ended with one state its work-state projection could name but not act on: observed, reinterpret. The response had arrived and been stored, and the process died before interpreting it. resume_call refused, correctly, because nothing in CodeAI yet ran an interpreter over stored bytes.

This chapter closes that state, and it rests on a distinction:

• A restart can tell us that the response has already arrived. It cannot reinterpret a response we failed to keep.
• Resumption requires facts. Interpretation is not one of them.

Facts, and what they are taken to mean

Pat Helland’s Immutability Changes Everything separates two kinds of entry in a ledger: observed facts, which are recorded and kept, and derived facts, which are calculated from them when needed (Helland, 2015).

His example is accounting. A received payment is observed; a balance is derived. Accountants don’t use erasers: a correction is a new entry, never an edit to an old one. Of a database’s transaction log he writes, “The truth is the log.” The database is a cache of a subset of it. He is equally blunt about the limit: having the right bits is not enough, because you still have to know how to interpret them.

Quinlan and Dorward’s Venti archival server gives the storage half of the argument (Quinlan and Dorward, 2002). Each block is addressed by a hash of its contents, so a block cannot be modified without changing its address, and storage is write-once by construction. The authors argue that a write-once policy greatly reduces the opportunities for data loss.

The AI evaluation literature has made the same choice for a different reason. HELM released all of its raw model prompts and completions publicly, for further analysis (Liang et al., 2023).

None of these papers is about model runtimes; the mapping is this chapter’s. For a CodeAI call it comes out as four layers:

Only the first row is a fact about the world. Everything below it is a conclusion, and conclusions are allowed to be wrong, provided the fact they came from is still there.

This is not a normalization chapter. Chapter 13 asked what usage numbers mean. This one asks what has to be true for any meaning to be revised later, after decisions have been built on it.

A defect found on the way

Before adding reinterpretation, the Stage 17 build read the existing projection interpret_attempt_as to see what it did when an observation’s bytes were gone. It did not refuse.

If the response file was missing or failed its hash check, it quietly fell back to a copy CodeAI had derived from the response at execution time. It then attached the observation’s artifact reference to the result anyway. A reader would see an interpretation naming bytes that did not exist.

That was fixed and committed on its own, before the feature, as CodeAI 5c37238. A missing or corrupt body now raises ObservationUnavailable. The regression test fails on the parent commit.

The derived copy is exactly the kind of thing this chapter warns about. It was an interpretation, and it was being presented as the observation.

What CodeAI now does

The difference between the two readings fits in a few lines. Interpreter v2 maps only explicit provider completion signals, per API dialect, and leaves everything else unknown. Reduced from CodeAI’s interpretation module:

_COMPLETION_REASONS = {                          # excerpt
    ("chat_completions", "stop"):              "complete",
    ("chat_completions", "length"):            "truncated",
    ("messages", "max_tokens"):                "truncated",
    ("responses", "max_output_tokens"):        "truncated",
    ...
}

# The two rules, reduced from interpret_v1 and interpret_v2:
v1_state = "complete" if output_text else "empty"                          # text means complete
v2_state = _COMPLETION_REASONS.get((protocol, finish_reason), "unknown")   # the provider's reason decides

A response with text and finish_reason: "length" is complete under the first rule and truncated under the second. Nothing about the bytes differs; only the reading does.

CodeAI a451e34 makes the second reading actionable after the fact with two operations. The day-two process in the experiment below made exactly this call:

runtime.reinterpret_call(CALL, interpreter_version=INTERPRETER_V2,
                         policy_version=ATTEMPT_POLICY_V2)

reinterpret_attempt reads the attempt.observed event and the bytes it names, and nothing else: no derived copy, no adapter-authored messages. It verifies the bytes against their recorded hash, applies the named interpreter, and appends one attempt.interpreted record, marked as recorded by reinterpretation and causally linked to the observation. It needs no completion record, so it works on an attempt whose process died after the response was stored, and it is idempotent per version.

reinterpret_call interprets every attempt and derives every decision before appending anything. Only then does it append any missing interpretations and one call.reinterpreted record, holding the new status and reason, the per-attempt decisions, the interpretation IDs and observation hashes it used, the prior status and the record that held it, and provider_effect: false. It never edits call.status_decided or starts an attempt, staying idempotent per version pair.

The work-state projection and acceptance validation now adopt the latest of call.status_decided and call.reinterpreted, and the projection says which one it used (status_basis). Neither operation ever contacts a provider. If the bytes cannot be read, both refuse without appending anything.

One preservation, many readings over time:

    flowchart TD
    B(["raw response bytes<br/><i>preserved once</i>"]) --> H["content hash<br/><i>verified before every reading</i>"]
    H --> A["immutable artifact<br/><i>never rewritten</i>"]
    A --> V1["parser v1<br/><i>text means complete</i>"]
    A --> V2["parser v2<br/><i>the provider's reason decides</i>"]
    V1 --> I1["interpretation: complete"]
    V2 --> I2["interpretation: truncated"]
    I1 -.->|"same bytes, new reading"| I2
    I2 --> R["call.reinterpreted<br/><i>appended; history never edited</i>"]
  

The experiment

The demonstration was preregistered at CodeAI a451e34, from a clean tree, before it ran. Each phase ran as a separate operating-system process, standing in for work done on different days. A synthetic provider appended an fsync’d line to its own receipt log for every request, so provider effects are counted outside the ledger, and outbound connections were refused throughout.

The response is a synthetic fixture. It reproduces Chapter 11’s defect class rather than its bytes: text, with finish_reason: "length". Interpreter v1 reproduces the historical rule (text means complete). Interpreter v2 lets the finish reason decide.

Day one, day two, day three

Day one. One provider request, one receipt, ten events. Interpreter v1 read the response as complete, the attempt decision recorded “v1: no error recorded”, call.status_decided recorded succeeded (“final attempt accepted”), and the work state projected the task’s next step as “check and accept”.

A counterfactual v2 projection, run in the same inspection, already said unresolved, and recorded nothing. The better answer was computable on day one, but it had no standing.

Day two. A different process opened the same files and called reinterpret_call under v2.

It appended two events: the first was a v2 interpretation, truncated for provider reason length, caused by the original attempt.observed event and naming body ff29c451…, while the second was call.reinterpreted, recording unresolved (“generation truncated, not treated as completed cognition”) with prior status succeeded.

The provider’s receipt log still held one line, and the response file still hashed to the value observed on day one, with the ten day-one events still first and unchanged. The v1 interpretation and the day-one decision are still there; they are no longer adopted.

Repeating the reinterpretation appended nothing and returned the same record.

Day three. A third process projected the work state. The call is unresolved, on the basis reinterpretation:attempt-interpretation-v2/attempt-policy-v2, and the task’s next step is no longer “check and accept” but “start the call”.

Then someone tried to accept the task anyway, citing the day-one v1 interpretation, with a check on the output text that passed. Acceptance was refused for two reasons: source_call_not_succeeded and interpretation_not_decision_basis. The first holds because the adopted status is now unresolved, while the second holds because the cited interpretation is no longer the one the adopted status rests on. The ledger ended at fifteen events.

Nothing was asked of the model after day one; what changed was what the system concluded from what it already had, and therefore what it would do next.

The attempt killed after the bytes arrived

The producer was killed with Popen.kill immediately after attempt.observed was committed. The ledger held five events and the receipt log one line.

A new process found the stage “observed”, with next operation “reinterpret”. The old projection, interpret_attempt_as, returned nothing, because it needs a completion record that was never written.

reinterpret_attempt read the preserved bytes and recorded truncated: one event appended, receipts still one. The work state moved from observed, reinterpret to interpreted, redecide.

That is the first post-effect state from Chapter 16’s table that CodeAI now advances automatically. Redeciding and finalizing are still not automated.

Without the bytes

Two copies of the day-one directory were damaged. In one the response file was deleted; in the other its bytes were altered and the recorded hash left alone.

Both the projection and the reinterpretation raised ObservationUnavailable. One said “response body bytes are missing”; the other said they “do not match their recorded sha256”. The ledgers stayed at ten events, and the receipt logs at one.

The runtime can still tell you that a response arrived, when, and what hash it had. It can no longer tell you what the response meant under a rule written after the fact. The only way to find out would be to ask the provider again, and that would be a different response.

The fourth copy, also with its body deleted, was read by CodeAI 68f4ba0, the commit before the defect fix, in a separate git worktree. It returned an interpretation, truncated for reason length, and named artifact ff29c451… as its evidence. The file with that hash was not on disk.

The conclusion happened to be right, because the derived copy it silently used still held the finish reason. That is the dangerous version of this failure. A wrong answer gets investigated. A right answer that names evidence which is not there passes review, and it stays wrong about its provenance for as long as anyone relies on it.

Checking it without trusting it

The bundle’s verifier imports neither CodeAI nor the producer. It reads the preserved response bytes and applies its own statement of both rules — under v1 text means complete, under v2 the finish reason decides — and checks the recorded conclusions against that reading: day one’s success, day two’s truncated and unresolved, and the counterfactual. It requires the v2 interpretation to be caused by the observation and to name the same bytes, the receipt count not to move during reinterpretation, and day one’s events to be an unchanged prefix of day three’s. It also checks the killed attempt, both refusals, the before-fix probe, the preregistration order, and the clean commit.

All 17 semantic claims pass. The full run, which adds byte hashes for 414 files, exits 0. A fresh CodeAI process reopened the ledgers and projected exactly the work states the original inspections recorded.

Five seeded corruptions were run with the byte inventory bypassed, so only meaning could catch them:

What this is not

• Not a proof that v2 is right. It is right about this defect class, on this fixture. v3 over the same bytes is the expected future, not a failure.
• Not undo. Anything done under the day-one conclusion stays done. A task accepted, a retry not taken, or an email sent under v1 is still in history. This run did not exercise a task that was already completed before its source call was reinterpreted.
• Not evidence of truth. A matching hash proves the bytes are the ones that were stored. It does not prove they are what the provider sent, or that what the model said is correct.
• Not normalization. The interpreter changed; the point is when it can change, and what it may and may not touch when it does.

Where it is still weak

1. Only preserved observations are reinterpreted. Adapter-authored error text is not an observation and is not used, so an attempt whose original interpretation relied on it may not replay identically.
2. Attempts without attempt.observed cannot be reinterpreted at all. That includes fake adapters and older records.
3. Latest record wins. There is no precedence between versions; reinterpreting with an older version afterwards would be adopted.
4. Redecide and finalize are not automated for interrupted attempts.
5. The derived copy still exists for attempts whose observation names no body.
6. A synthetic provider, one defect class, one fixture, a single writer.

Do this now

Thirty minutes. Find out whether you could change your mind about yesterday.

1. Pick one model call your system made yesterday. Find what it stored. Is it the provider’s response bytes, or only your extracted text, status and token counts?
2. Write down one rule your code applies to responses: success detection, JSON extraction, refusal detection, usage accounting. Suppose it was wrong. Could you re-run a corrected rule over yesterday’s calls without calling the model again?
3. Find where the result of that rule is stored. When you fix the rule, does the old result get overwritten, or does a new, versioned result sit beside it?
4. Find one downstream thing that trusted the old result: a status, a retry that didn’t happen, a merged change. Would it notice?

If you are building with an assistant:

Make model-call interpretations revisable without repeating the call.
- Store the provider's response bytes, content-addressed, before extracting
  anything. The observation record names the bytes and their hash.
- Record every interpretation with an interpreter version, linked to the
  observation it read. Record decisions with a policy version.
- Add reinterpret(call, interpreter_version, policy_version): verify the
  bytes' hash, compute every new interpretation and decision first, then
  append them plus one "call reinterpreted" record naming the prior status.
  Never edit earlier records. Idempotent per version pair. Never call the
  provider.
- If the bytes are missing or fail their hash, refuse and append nothing.
  Never fall back to a derived copy while naming the original bytes.
- Make status projections and acceptance use the latest adopted record.
- Test across separate processes: record under the old rule, reinterpret
  under the new one, and show provider request count unchanged, bytes
  unchanged, old records intact, next step changed. Delete the bytes on a
  copy and show that reinterpretation is refused.

Failure modes

• Storing only the interpretation. When the rule turns out to be wrong, there is nothing to apply the corrected rule to.
• Asking the model again. It returns a different response, costs money again, and erases the question of what was received when the decision was made.
• Overwriting the old conclusion. You lose what decisions were actually based on.
• A better interpretation with no standing. Chapter 13’s projections were correct, and nothing acted on them.
• Substituting a derived copy for missing bytes. The conclusion may even be right. Its provenance is false.
• Trusting a hash as truth. It proves the bytes are unchanged, not that the content is correct.
• Assuming reinterpretation undoes effects. It changes what the system concludes next, not what it already did.

What this chapter established

• An observation must outlive its interpretation. Interpretations are derived facts and are allowed to be wrong; recovering from that requires the observed fact they came from. Helland’s observed and derived facts, Venti’s write-once content addressing and HELM’s release of raw completions support the general reasoning. The mapping is this chapter’s.
• CodeAI 5c37238 refuses, instead of silently substituting a derived copy, when a response’s bytes are missing or corrupt; a451e34 adds reinterpret_attempt and reinterpret_call, which read only preserved observations and append versioned interpretations and a call.reinterpreted record without editing history, with status projections and acceptance adopting the latest record.
• In separate processes, a truncated response recorded as a success on day one was reinterpreted on day two as unresolved, with no new provider requests while bytes and day-one history stayed unchanged; the task’s next step changed from “check and accept” to “start the call”, and an acceptance citing the old interpretation was refused.
• An attempt killed after its response was stored was advanced from the bytes alone; with the bytes deleted or corrupted, reinterpretation was refused and nothing was appended; and the code before the fix returned an interpretation naming bytes that no longer existed.
• An independent verifier passed 17 semantic claims plus byte hashes for 414 files, and rejected five seeded corruptions.
• Not established: that v2 is correct in general, undoing effects made under an old conclusion, version precedence, or automated redecide and finalize.

Next

An interpretation is still one conclusion about one response, held by the runtime. A review that says a claim is unsupported is a response, too. So is the justification for accepting a task.

The next step is to break conclusions into claims that name their evidence, and to bind every decision to the claims, and the versions of their interpretation, that it rested on. Then, when a reading changes, the system can say which decisions stood on it.

Continue with Claims, Evidence, and Decisions.

References

• Pat Helland. Immutability Changes Everything. 7th Biennial Conference on Innovative Data Systems Research (CIDR), 2015. https://www.cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
• Percy Liang, Rishi Bommasani, Tony Lee, et al. Holistic Evaluation of Language Models. Transactions on Machine Learning Research, 2023. arXiv:2211.09110. https://arxiv.org/abs/2211.09110
• Sean Quinlan and Sean Dorward. Venti: A New Approach to Archival Storage. Proceedings of the FAST 2002 Conference on File and Storage Technologies, USENIX, 2002. https://www.usenix.org/legacy/publications/library/proceedings/fast02/quinlan/quinlan.pdf

Implementation sources:

• CodeAI 5c37238: src/codeai/interpretation.py (ObservationUnavailable); src/codeai/runtime.py (interpret_attempt_as refuses instead of substituting); regression test in tests/test_interpretation.py.
• CodeAI a451e34:
  • src/codeai/runtime.py: reinterpret_attempt, reinterpret_call, _interpret_from_observation, _append_reinterpretation, _latest_status_record.
  • src/codeai/providers.py: output_text_for.
  • src/codeai/interpretation.py: _COMPLETION_REASONS, interpret_v1, interpret_v2 (the reduced rules and map excerpt above; inspected in current source).
  • src/codeai/workstate.py: adopted status with status_basis.
  • src/codeai/acceptance.py: adopted status in _validate_source.
  • Tests: tests/test_raw_output.py (9); full suite 327 passed.
• Evidence: experiments/applied-ai/evidence/raw-output/2026-09-13-a451e34/.
  • The preregistration and execution record.
  • Five cases, each with an SQLite ledger, artifact store, provider receipt log, per-day inspections and exported events.
  • The independent verify.py, five seeded corruptions, test outputs, chapter-evidence-report.md and hashes.json.
• Producer: experiments/applied-ai/raw_output_demo.py. The executed copy is pinned as the bundle’s run.py.