Part 3 — Give Intelligence a Runtime

The step that didn’t finish

A process was reviewing a paragraph. It had compiled its context, recorded the call’s manifest, and sent the request. While the provider was still working on it, the process died.

The standard recovery is familiar: restart, find the step that didn’t finish, and run it again. Someone did exactly that in this chapter’s experiment, on a copy of the dead process’s files, and the step re-ran to completion.

The provider’s own receipt log then showed two requests, byte-for-byte the same request under the same idempotency key, from two different processes. The work had been paid for twice.

The ledger had not been silent. Before the crash, it had recorded that an attempt started with nothing observed afterwards, a meaning the restart never asked about.

Can another process continue the work without reconstructing what happened from human memory?

Restart is not resume

Chapter 15 ended with a narrower version of this problem. A package could be recovered after a clean exit, but a later process could not see what the package had been selected from, or which compilations had failed.

Two different capabilities hide behind the word “recovery”.

Restart means the durable state survives the process. The ledger is still there, the artifacts are intact, and another process can open them. CodeAI has had that since Chapter 11.

Resume means another process can decide what to do next from what is recorded, without asking the person who was watching. It has to answer seven questions:

A database that survives a crash answers none of these by itself. They are questions about the work, not the storage.

There is a related idea that is easy to confuse with this one. MemGPT treats a model’s limited context like an operating system treats memory, moving information between fast and slow tiers so the model can work with more than fits in its window (Packer et al., 2023). That is memory for the model: what it can draw on while it thinks. This chapter is about memory for the process: whether the system knows what it was doing when it stopped. A model with excellent long-term memory can still be run twice by a process that doesn’t.

Most of the record already existed

Earlier chapters had built most of the trail, one boundary at a time. Chapter 11 writes intent (call.manifest) before any attempt and records each attempt, and the interpretation work after it records interpretations and attempt decisions. Chapter 12 preserves the transport observation. Chapter 14 records acceptance and completion. And Stage 15B, built after Chapter 15, lets the selected context reach the request for calls that opt in, binding its rendered bytes to the manifest before any effect (experiments/applied-ai/evidence/context-rendering/2026-09-13-0f9a83b/).

Two gaps remained. Chapter 15 had found that the offered inventory and failed compilations lived only in the experiment harness. And nothing turned the trail into an answer: a later process could read every event and still have to work out for itself what they meant.

Stage 16 closes both. The next section is about the one distinction that decides everything else.

Effects that cannot be taken back

Elnozahy, Alvisi, Wang and Johnson’s survey of rollback recovery separates a system from what it calls the outside world (Elnozahy et al., 2002). Processes inside the system can be rolled back and replayed. The outside world cannot: in their examples, a printer cannot unprint a character, and a cash machine cannot take back money it has dispensed. In their words, the outside world cannot be relied on to roll back.

That produces what they call the output commit problem. Before a system sends something to the outside world, it must make sure the state it sent from will survive any later failure. Otherwise, after recovery, the system may send it again, or behave as if it never did.

A model provider is outside the world of a CodeAI process. The mapping is this chapter’s, not theirs. A served request has already cost money and produced a generation, whether or not the response ever arrives. No recovery on this side takes it back.

Lampson’s hints for system design state the same requirement from the logging side (Lampson, 1983). Log updates so the log records the truth about an object’s state, in entries that can be re-executed. Make actions atomic or restartable, where a restartable action can be partially executed any number of times without changing the result. His example: storing a set of values is restartable; adding one to a variable is not.

A provider call is an increment, not a store. Each execution is a new request, new spend, and a new generation.

So the question that governs resume is not “did this step finish?” It is: could the effect have happened? CodeAI’s call path answers it with an ordering chosen in Chapter 11. attempt.started is committed to the ledger before the request is sent, and the observation is committed after the response arrives. Between those two events, the honest answer is “unknown”.

The projection

CodeAI now has Runtime.work_state(task_id). It is a pure projection: it reads the ledger and appends nothing. Each call is classified by the last evidence recorded for it:

The table is a ladder in code: each rung asks whether a particular event exists. Reduced from the current source, with the reason strings shortened:

if "call.completed" in kinds:
    stage, next_op = "completed", NONE
elif "call.preparation_failed" in kinds:
    stage, next_op = "preparation_failed", FIX_INPUTS
elif "call.manifest" not in kinds:
    stage, next_op = "requested", START_CALL           # no manifest, so no attempt can have started
elif not attempts:
    stage, next_op = "manifest_recorded", START_CALL   # manifest recorded, no attempt: no provider effect
else:
    seen = kinds_for(last_attempt)
    if not seen & {"attempt.observed", "attempt.interpreted",
                   "attempt.retry_decided", "attempt.completed"}:
        stage, next_op = "effect_unknown", RECONCILE_EFFECT
        provider_effect, duplicate_effect_risk = "unknown", True
    elif "attempt.interpreted" not in seen:
        stage, next_op = "observed", REINTERPRET
    ...

It derives the stage solely from which events are present in the ledger, using no clock, provider query, or memory held outside it. The dangerous rung is the fifth: an attempt that started and left no trace after it. The projection does not guess whether the provider served it; it records “unknown” and a duplicate risk, and that is what resume_call refuses to override.

Compilations get the same treatment. A compilation that was requested and never finished projects “recompile”, which is always safe because compiling has no external effect. A compilation that failed projects “fix the inputs”, together with the inventory it was offered and any required IDs that were missing.

The task gets a next operation too. That’s either the first unresolved call’s, or, once a call has succeeded, Chapter 14’s “check and accept”.

Runtime.resume_call(call_id) acts on exactly one row: the call whose record proves no provider effect was possible. Before acting, if a manifest was recorded, it rebuilds the request and checks three hashes against the manifest: the context package identity, the rendered context and the request body. If any differs, it refuses. The call it would send is no longer the call that was intended. Otherwise it appends call.resumed, linking the old call to a new call with the same idempotency key, and runs the new call.

For every other state it raises ResumeRefused, names the projected next operation, and appends nothing.

The handoff the chapter is about — one process dies, another continues from the record alone:

    flowchart LR
    subgraph PA["process A"]
        direction TB
        A1["append intent<br/>before effect"] --> A2["act"] --> A3["append observation"]
    end
    LED[("append-only ledger")]
    A1 --> LED
    A2 --> LED
    A3 --> LED
    LED -.->|"process A dies"| PB["process B<br/><i>reopens the same ledger</i>"]
    PB --> PR["work-state projection<br/><i>reads only, appends nothing</i>"]
    PR --> D{"effect proven<br/>absent?"}
    D -->|"yes"| RS["resume<br/><i>same key, verified intent</i>"]
    D -->|"no"| RF["refuse<br/><i>reconcile first, never retry blind</i>"]
  

The experiment

The demonstration was preregistered at CodeAI 68f4ba0, from a clean tree, before it ran. Every phase ran as a separate operating-system process over one case directory: producing, inspecting, resuming, and restarting naively.

Two details make the evidence stronger than a test. Producers were really killed, with Popen.kill, which runs no cleanup, at durable checkpoints: immediately after a named event was committed, or while the provider held the request. And provider effects were counted outside the ledger: the synthetic provider appended an fsync’d line to its own receipt log for every request it received, so the ledger’s claims can be checked against something it didn’t write.

Outbound network connections were refused throughout, and no real provider was called.

Resume once, refuse twice, and a restart that paid twice

Resumed. The producer was killed right after call.manifest, with 7 events in the ledger and nothing in the provider’s log. A new process projected “manifest recorded, no provider effect, start the call”. A third process called resume_call, rebuilt the request, found the same request body hash the manifest had recorded, and appended call.resumed from call-16 to a new call under key-16 before running it.

Afterwards the provider’s log held exactly one receipt, whose body hash equals the hash in both manifests. The old call now projects as superseded by the new one, the new call is completed, and the task’s next operation has moved from “start the call” to “check and accept”. The ledger grew to 17 events. Nobody had to remember which step was pending.

Refused for drift. The same interruption, resumed with a different model configured, was refused before anything was sent, because the recorded intent no longer reproduces its request body hash. The ledger stayed at 7 events and the provider saw nothing. Resuming does not mean running whatever the code would send today. It means running what was recorded as intended, or refusing.

Refused for an unknown effect. The producer was killed while the provider held the request, with one receipt already in the provider’s log and only attempt.started in the ledger. A new process projected “effect unknown, reconcile the effect, repeating risks a duplicate”. resume_call refused, the ledger stayed at 8 events, and the receipt count stayed at 1.

The naive restart. A copy of that same directory was handed to a process that did what restart loops usually do: take the recorded request and issue it again without consulting the work state. It succeeded, and the provider’s log now shows two receipts with identical request bodies, from two processes, under the same idempotency key.

The idempotency key didn’t help, because replay only matches calls that completed, and this one never had. The ledger does record both calls, so the duplicate is visible afterwards, but nothing prevented it.

Observed, not automated. The producer was killed after the observation was committed. The projection says “reinterpret”: the response bytes are preserved, so the provider is not needed again. resume_call refused this too, for a different reason. It isn’t unsafe; CodeAI simply doesn’t yet automate re-running the interpreter over stored bytes. The distinction is recorded, not blurred.

Compilations that stopped or failed

Killed after context.compilation_requested, the ledger held 4 events: the task, the objective, the claim, and the compilation request. A new process recovered what had been offered (A, B and D, with their required flags, sizes and lineage) and projected “recompile”.

The compilation that failed on a missing required input was recorded, not just raised. The failure event names RequiredContextMissing. The inventory shows A (required, estimated size 12), B (required, size 0) and D (optional, size 0), and the projection derives the missing required ID, never-offered, from the difference between what was required and what was offered. Chapter 15 found exactly this information missing from the runtime.

Checking it without trusting it

The bundle’s verifier imports neither CodeAI nor the producer. It holds its own implementation of the classification rules and re-derives every call and compilation state from the exported events. It then compares those with the states each inspecting process recorded. It also checks that every inspection appended nothing, and that every provider receipt is covered by an attempt.started in the ledger — an effect with no recorded intent to cause it would fail.

All 73 of its claims pass, including byte hashes for 424 files. Separately, a fresh CodeAI process reopened three of the ledgers and projected exactly the states the original inspections had recorded.

Five seeded corruptions were each run with the byte inventory bypassed, so only meaning could catch them:

What this is not

• Not crash atomicity. Every kill happened at a durable checkpoint — after a commit, or inside the transport — which leaves dying between an artifact write and its event, or during a database commit, still uncovered.
• Not exactly-once. A provider effect whose outcome is unknown is refused rather than resolved, since CodeAI has no way to ask the provider whether a request was served and no way to record a reconciliation decision.
• Not concurrency. Two processes resuming the same call at the same moment could both start it.
• Not automated recovery of post-effect states. Reinterpret, redecide, start the next attempt and finalize are named precisely, and left to the next chapters.

Where it is still weak

1. One state resumes automatically. Only calls with no attempt started are resumed.
2. No reconciliation. “Reconcile the effect” names the problem without a mechanism.
3. Kills only at durable checkpoints.
4. Single writer.
5. No drift check when resuming from requested. With no manifest recorded, there is nothing to compare against.
6. Supersession is by idempotency key only.
7. Inventory sizes include estimates. A’s recorded size 12 is estimated from its serialized payload, and artifacts and claims count zero.
8. Fanout compilations are not recorded through this path.
9. A synthetic provider, and one fixture.

Do this now

Forty minutes. Find out whether your system resumes or merely restarts.

1. Pick a pipeline or agent loop that calls a model. Kill it while a model request is in flight, restart it, and count the requests on the provider’s side: its usage page, logs or bill. Was the work done twice?
2. For each step, ask whether a later process can distinguish “the effect may have happened” from “the effect did not happen”. If the record doesn’t mark the point just before the effect, the answer is no.
3. List every effect outside your process: model calls, emails, payments, tickets, file writes. Mark each as restartable (setting a value) or not (an increment). The second kind must never be retried on a guess.
4. Find the code that decides what to do after a restart. Does it read a record written before the crash, or does it re-run whatever “didn’t finish”?

If you are building with an assistant:

Make a model-calling workflow resumable, not just restartable.
- Record intent before any effect: the full request specification, and for
  context, the offered inventory with requirements, sizes and lineage.
  Record failures of compilation and preparation as events.
- Append "attempt started" durably before sending a request and "observed"
  after the response is stored. Between them the effect is unknown.
- Add a pure projection that classifies each operation by its last recorded
  evidence and names the next operation: start (no effect possible),
  reconcile (effect unknown; never repeat automatically), reinterpret,
  redecide, finalize, fix inputs, recompile.
- Resume only operations with no recorded attempt. Re-issue under the same
  idempotency key as a new, linked operation, and refuse if the rebuilt
  request no longer matches the recorded intent.
- Test with separate processes and real kills at each boundary. Count
  provider effects with a log the provider writes, not the ledger. Include a
  naive restart on a copy and show that it duplicates the effect.

Failure modes

• Treating restart as recovery. The files survived; the process still didn’t know what it was doing.
• Re-running the step that didn’t finish. A served request is an increment, not a store.
• Trusting the idempotency key to catch it. Replay only matches operations that completed.
• No marker before the effect. Without it, “didn’t happen” and “unknown” look the same.
• Resuming what the code would send today. Resume what was recorded as intended, or refuse.
• Recording successes only. Failed compilations and preparations are part of the working state.
• Confusing model memory with process memory. A model that remembers is not a system that knows what it did.
• Claiming exactly-once. An unknown effect has been refused, not resolved.

What this chapter established

• Restart ≠ resume. Resuming requires answering, from recorded facts, what was attempted, from which inputs, what was required, what failed, what succeeded, what is unresolved and what is safe next.
• The deciding question is whether an effect could have happened. A model provider is outside the process; a served request cannot be taken back (Elnozahy et al.), and a provider call is not a restartable action (Lampson). The mapping is this chapter’s.
• CodeAI 68f4ba0 records compilation with its offered inventory and failures, projects each operation’s next step and whether an effect may have happened, and resumes only where no effect was possible, refusing when the recorded intent no longer reproduces.
• In separate processes with real kills, one resume after the manifest produced exactly one provider receipt with the recorded request hash; the same resume with a different model was refused; and with the effect unknown, resume was refused while a naive restart of an identical copy produced a second, identical provider request.
• After observation, resume was refused with “reinterpret”, and interrupted and failed compilations were recoverable with their inventories.
• A verifier sharing no code with CodeAI re-derived every state from exported events, confirmed every receipt had a recorded attempt, passed 73 claims including byte hashes, and rejected five seeded corruptions.
• Not established: crash atomicity, exactly-once effects, concurrent resume, reconciliation, or automated recovery after an effect.

Next

Four of the states the projection names — reinterpret, redecide, start the next attempt, finalize — share a premise. The provider’s response still exists, exactly as it arrived. Reinterpreting is only safe because the bytes were preserved before anything was concluded from them. Automating it means running a versioned interpreter over stored observations, without replacing them.

Continue with Raw Output First.

References

• E. N. (Mootaz) Elnozahy, Lorenzo Alvisi, Yi-Min Wang, and David B. Johnson. A Survey of Rollback-Recovery Protocols in Message-Passing Systems. ACM Computing Surveys, 2002. https://doi.org/10.1145/568522.568525
• Butler W. Lampson. Hints for Computer System Design. Proceedings of the Ninth ACM Symposium on Operating Systems Principles, Operating Systems Review 17(5), 1983, pp. 33–48. https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/acrobat-17.pdf
• Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560, 2023. https://arxiv.org/abs/2310.08560

Implementation sources: CodeAI 68f4ba0. src/codeai/workstate.py: WORK_STATE_V1, project_work_state, NextOperation, CallState, CompilationState, resume_call, spec_from_payload, ResumeRefused. src/codeai/context.py: ContextCompiler.offered_candidates, extracted unchanged from compile_with_trace. src/codeai/runtime.py: compile_and_record_context (context.compilation_requested, context.compiled, context.compilation_failed), Runtime.work_state, Runtime.resume_call. Stage 15B rendering: src/codeai/rendering.py at 0f9a83b. Tests: tests/test_work_state.py (14); full suite 317 passed. Evidence: experiments/applied-ai/evidence/working-state/2026-09-13-68f4ba0/, containing the preregistration and execution record, seven cases plus the naive-restart copy (each with termination record, checkpoint, SQLite ledger, provider receipt log, inspections before and after actions, and exported events), the independent verify.py, five seeded corruptions, test outputs, chapter-evidence-report.md and hashes.json. Producer: experiments/applied-ai/working_state_demo.py (the executed copy is pinned as the bundle’s run.py).