Part 4 — Make It Safe and Verifiable

The line that may already be there

Constructed scene. An adapter appends one line to a file. The process dies before the completion is recorded. On restart the ledger shows no completed result — but the file has changed.

Three responses are available, and two of them lie. Appending again may write the line twice. Writing a success record by hand manufactures evidence for an effect nobody observed. Saying “unknown” is the only honest answer, and it is also the least actionable one. The question that matters is narrower than “what do we do?” It is: what evidence would justify another append?

Chapter 21 ended with failure inviting another attempt. This chapter controls the invitation. A retry is not a loop. It is a claim about the first attempt — what it intended and observed, whether it was authorized, what was checked — and a decision about what may happen to the world next.

When may the runtime repeat work without duplicating an effect?

Retry, replay, duplicate

The distinction is retry ≠ replay ≠ duplicate.

An idempotency key alone does not establish which of these is safe. A key is a lookup handle, not the identity of the intended operation. The chapter’s operational question is therefore: what evidence justifies returning a prior result instead of taking another effect?

The distributed-systems literature gives this question its shape. RIFL converts at-least-once RPCs into exactly-once ones by durably recording completed results and returning the saved result when a retry arrives, and it names four problems every such mechanism must solve: RPC identification, completion-record durability, retry rendezvous, and garbage collection (Lee et al., 2015). Its hardest requirement is that the completion record be created atomically with the operation’s mutations: in the paper’s words, “without a visible completion record, or vice versa.” That atomicity is exactly what CodeAI’s file-plus-ledger path cannot provide — the adapter writes wherever it likes, and the ledger hears about it afterwards. The mapping of RIFL’s four problems onto CodeAI’s action path is the book’s own, and the gap is the point: a key lookup is one quarter of RIFL, not RIFL.

Birrell and Nelson’s RPC design is the older foundation: remote calls need retransmission because responses get lost, and a timeout says nothing about whether the far side acted (Birrell and Nelson, 1984). That supports separating the logical request from its attempts — CodeAI’s Chapters 11 and 16 already do — but retransmission alone is at-least-once by construction. It does not, by itself, tell a retry from a duplicate. The application to CodeAI’s replay rule is again the book’s mapping, not the paper’s result.

ARIES is the deliberate contrast, used lightly. Write-ahead logging recovers database transactions because the log is the recovery path for pages the system manages (Mohan et al., 1992). Arbitrary file writes and external API effects live outside those managed pages. No logging discipline around them manufactures atomicity. The chapter borrows ARIES’ vocabulary — completion records, recovery obligations — and none of its guarantees.

What CodeAI does

The source-inspected execute_action order on the fresh path is: append action.requested; look up a prior completion by idempotency key; convert the capability and call policy.require; compare a supplied precondition against the current reading when both are available; call adapter.execute; read the state resolver again and finalize; append action.completed. ¹

The replay branch in current CodeAI (a1b562a) is no longer a bare key lookup. Its order is: authority first, identity second, replay last. ²

Authority first. The current request’s capability is checked against the currently supplied grant before anything recorded is consulted. A denied replay records an action.completed with status DENIED for the new request — the same durable shape as a fresh denial — and returns it. A denied caller receives only the denial, with no existence oracle and no fingerprint dimensions exposed. Ability to replay an old record is not authority to request the operation now. ²

Identity second. The incoming request’s fingerprint is compared against the fingerprint rebuilt from the original request’s recorded action.requested payload, so the check reproduces after ledger reopen. On mismatch the runtime appends action.replay_refused — naming the key, the original action, the fingerprint version, and the mismatching dimensions — and raises IdempotencyConflictError before any effect. A conflict leaves durable evidence, not only an in-memory exception; the bare raise would have left action.requested unresolved, the same class of gap Chapter 21 closed for checks. ³

Replay last. Only the same key plus the same identity returns the recorded completion, relabeled with the new action ID and reused_from_action_id pointing at the original physical effect. Whatever the first execution recorded — success, failure, even denial — is what replay returns. ²

The fingerprint, action-fingerprint-v1, is deliberately narrower than the request and wider than the key: ⁴

The call path is the architectural precedent, and the comparison is instructive. Call replay looks up call.completed events that carry a matching call.manifest — legacy completions without terminal evidence are misses — then _check_replay_fingerprint compares prompt hash, chamber, requested model, and rendered context, raising IdempotencyConflictError naming the dimensions before any provider effect. An identical re-request replays with replayed: true and the original call ID. ⁵

Two differences matter. First, the call conflict stays in memory: call.requested is appended, the check raises, and no durable conflict event follows. The action path now records its refusals; the call path’s gap is noted, not repaired here. Second, calls carry a manifest — a runtime-built record of what was actually sent — while actions have only the request the caller supplied. The action fingerprint is therefore weaker by construction: it binds the requested operation, not an independently built manifest of the performed one. ⁶

The precondition rule, stated plainly

Precondition is part of request identity, and it is not re-evaluated before returning a recorded completion. Both halves need their reason on the page.

Identity, because the precondition names the world the operation was planned against. A request planned against state A and a request planned against state C are different planning instances even when their instructions match; conflating them lets one worldview borrow another’s outcome.

No re-evaluation, because re-checking would destroy legitimate idempotent replay. The original precondition was state A; the action moved the world to B; the same request arrives again. If replay required the current state to still equal A, then a successful action would destroy the condition required to replay its own result — the mechanism would fail exactly when it worked. The current-state comparison belongs to fresh execution, where it refuses drift before acting. Replay answers a different question — “what did this operation instance record?” — and the recorded precondition is part of how the instance is identified, not a gate on reading its history. ⁷

The cost of that choice is explicit: if the world moved back to A after a drift failure, the same key still replays the old FAILED rather than re-evaluating. A changed context that should produce a new decision needs a new key. Keys scope one logical instance and its recorded outcome; they are not subscriptions to the world.

The executed demo, read exactly

The preserved retry-effects demo runs five offline cases with fake adapters, and its independent verifier checks the summary plus a seeded corruption. ⁸

Case B is the defect this chapter’s revision addresses, and the evidence discipline matters here: the demo ran against the old code, so it establishes that the defect existed, not that the repair works. The repair is source-inspected and regression-tested in current CodeAI — and the pinned rerun below replays every demo case against the repaired semantics with a ledger-based verifier. Historical demo found the defect; current source contains the repair; regression tests exercise it; the rerun measures it.⁸

Under the revision, case B’s two variants separate cleanly. Same key with a different operation and proper authority → IdempotencyConflictError naming the dimension, action.replay_refused on the ledger, zero effects. Same key with the same operation but no authority → DENIED completion, zero effects, no information about the recorded operation. Both were executed as teaching fragments against the working tree; neither is a pinned stage run. ²

Failure is cached, and that is a decision

Case D deserves its own section because it breaks the most natural retry loop. The adapter wrote the file and then raised. The ledger says FAILED. A naive retry — catch the exception, call again with a fresh key — writes the file twice. A same-key retry replays the FAILED without touching the file. Both behaviors are now covered, and neither recovers anything: ⁸

write_file()
raise RuntimeError("connection lost")

Constructed fragment using actual APIs, executed against the working tree as teaching code. The file marker appears once; the ledger says failed; the retry returns the failure with reused_from_action_id set and the adapter untouched. ¹

first = runtime.execute_action(req("d1", "kc"), authority=auth, adapter=crasher)
second = runtime.execute_action(req("d2", "kc"), authority=auth, adapter=crasher)
assert (first.status, second.status) == (ActionStatus.FAILED, ActionStatus.FAILED)
assert second.reused_from_action_id == "d1"
assert crasher.calls == 1

This gives the chapter’s hardest-won distinction: operation reported failed ≠ effect did not happen. Replaying a recorded failure says “this request already produced this recorded outcome.” It returns history without attempting recovery, and Chapter 19’s observed state on the FAILED result remains the only witness to what the world looked like — a reading taken after the raise, not during it.

Where the cache stops

Caching failure protects only failures the ledger hears about, and that boundary was narrower than case D made it look.

Case D’s adapter raised RuntimeError, and at a1b562a that was the only adapter exception execute_action turned into a recorded FAILED. Real transport failures are usually something else: ConnectionError and TimeoutError are OSError subclasses. A probe during editing gave an adapter that writes a marker and then raises ConnectionError. The exception escaped, the ledger held action.requested with no completion, and a same-key retry found no record to replay — so it called the adapter again. Two adapter calls, two markers, the exact duplicate this chapter exists to prevent. ¹

The repair, uncommitted in the CodeAI working tree, is one clause: any adapter Exception after the authority and precondition checks is recorded as FAILED with the runtime’s observation. A regression test repeats the probe’s shape and asserts one adapter call, two FAILED completions, and a replay link. Rerunning the probe after the change gives one call and one marker. Neither the probe nor the test is a pinned stage run, and the historical demo is unchanged. ⁹

What the repair cannot reach is the chapter’s opening scene. If the process dies between the adapter’s write and the completion append, no exception handler runs. The ledger holds an orphan action.requested, and replay lookup reads only action.completed events — so a same-key retry sees an empty lookup and acts again. The idempotency key protects outcomes that reached the ledger. It does not protect effects whose outcome never did, and nothing in CodeAI currently projects orphaned action requests the way Chapter 16 projects orphaned call attempts. ¹⁰

The two timelines, side by side — recorded completion replays, orphaned request repeats:

    sequenceDiagram
    participant Caller
    participant Runtime
    participant Adapter
    participant World
    participant Ledger
    Caller->>Runtime: execute_action (key K)
    Runtime->>Ledger: append action.requested
    Runtime->>Adapter: execute
    Adapter->>World: effect happens
    alt completion recorded
        Adapter->>Runtime: return result
        Runtime->>Ledger: append action.completed
        Caller->>Runtime: retry same key K
        Runtime->>Ledger: lookup K → replay, no new effect
    else crash before the record
        Adapter--xRuntime: process dies
        Caller->>Runtime: retry same key K
        Runtime->>Ledger: lookup K → nothing
        Runtime->>Adapter: execute again
        Adapter->>World: effect happens twice
    end
  

What CodeAI cannot do here is as important as what it can. Chapter 16 built resume_call with RECONCILE_EFFECT for provider calls: an attempt started with no observation projects as effect-unknown, and resume is refused except where the record proves no effect could have happened. That machinery is call-only. There is no action-level projection of working state, no action resume, and no RECONCILE_EFFECT path for actions — reconciling the file with the FAILED record remains an operator judgment outside the runtime. The demo’s note says exactly this, and the chapter does not improve on it. ^{11 12}

Two threads, one key, two effects

Sequential duplicate safety does not establish concurrent safety, and the ledger shows why: appends commit individually, the events table constrains only event_id uniqueness, and no lock, key uniqueness, or compare-and-set spans the lookup-to-append interval. Two requests can both find no completion, both execute, and both record. ¹³

A bounded threaded probe confirms the window is real, not theoretical: two threads submitting the same key against one ledger file, three runs, two physical effects and two successes every time. An earlier variant of the probe, with threads opening the ledger simultaneously, failed even sooner with database is locked — concurrent writers are unsupported at the connection level here, not merely racy. That probe is now a frozen diagnostic inside the pinned rerun: barrier-synced, race window held open, 3 runs with 6 submissions and 6 effects preserved, repaired nothing. The durable claim remains the source-inspected absence of coordination, and the chapter claims nothing about concurrent safety. Any future concurrency mechanism will be measured against those six effects.¹³

Checking it without trusting it

The demo’s independent verifier imports no CodeAI code. It requires A at 2 requests / 1 effect with the reuse link, B replaying silently (asserted as the limitation, not as success), C raising with prompt_hash named and zero effects, D caching the failure with zero new effects, and E rejecting drift — plus a mutated copy claiming two physical effects, which it must reject. ¹⁴

Its limit mirrors the earlier chapters’ verifiers: it checks the producer’s summary fields, not the ledger. A summary that miscounted effects would pass if it miscounted consistently. The pinned rerun in experiments/applied-ai/evidence/retry-rerun/2026-09-14-1b3c7a2/ answers exactly that weakness: frozen protocol, no fixes during the run, and a stdlib-only verifier that recomputes from durable evidence — request counts, invocation receipts, marker files, fingerprints, keys, authority decisions, replay provenance, original and replayed IDs, reopen identity — with five seeded corruptions that alter replay provenance, inflate a receipt, drop the refusal or a request, or rewrite a FAILED completion, each rejected with the failure named.

The rerun covers exact sequential duplicates (2 requests, 1 effect, reuse link resolves); key collisions with zero second effects and the preserved conflict naming instruction; authority changes that return a DENIED completion with no reuse and no new effect; duplicates across process reopen; crash-gap cases for both exception families showing what the record establishes (FAILED replay, one effect) and what it cannot (whether the bytes changed is witnessed only by the marker file, and recovery stays operator judgment); precondition drift refused before effect; and a diagnostic concurrency probe — barrier-synced threads with the race window held open, recorded without repair: 3 runs, 6 submissions, 6 effects, every row reused=None. That probe is the frozen baseline the concurrency work must improve on.

What this is not

• Not exactly-once. Sequential replay suppresses known duplicates; concurrent submissions demonstrably duplicate, and crash-gap effects stay unknown rather than counted.
• Not transactional. No atomicity spans adapter effect and ledger append — RIFL’s central requirement is absent, not approximated.
• Retry policy lives elsewhere. The runtime returns records and refuses conflicts; deciding when trying again is wise belongs to policy over the evidence Chapters 18–21 preserve.
• Completion records accumulate. RIFL’s fourth problem — leases, acknowledgments, reclamation — has no counterpart here.
• A request binding, not a manifest. The fingerprint binds the requested operation, not an independently built record of what was performed.

Where it is still weak

1. Concurrent duplicates are unprotected. No lock, no key constraint, no compare-and-set spans the check-to-act interval; the probe shows two effects from two threads. ¹³
2. Completion and effect are not atomic. The adapter acts and the ledger hears later. An adapter exception is now recorded and replayed as FAILED, while a process death leaves an orphan action.requested that replay lookup ignores, so a same-key retry acts again. ¹⁵
3. Actions have no recovery projection. resume_call and RECONCILE_EFFECT serve calls only; an action FAILED-with-effect ends at operator judgment. ¹⁶
4. Call conflicts leave no durable record. Action refusals append action.replay_refused, whereas call fingerprint conflicts raise with only call.requested behind them. ¹⁷
5. Completion records are never reclaimed. Every replayable outcome lives forever; no acknowledgment, lease, or retention rule exists. ¹⁰
6. A missing current state still skips the precondition comparison on the fresh path. “Precondition supplied” remains weaker than “precondition enforced.” ¹
7. DENIED outcomes replay, by key-scoping rule. A caller that gains authority must use a new key for a new decision; the old key returns the recorded refusal, which can surprise. ²
8. The measured rerun has run; the architectural gaps it froze remain. Revision plus regression tests plus the pinned rerun establish the sequential semantics, while concurrency safety and crash-gap recovery stay unbuilt by design, awaiting reconciliation first.

Do this now

Thirty minutes. Retry something that already ran, and prove you did not do it twice.

1. Run an effectful action with a fixed key and a counting adapter. Submit the identical request again and confirm one effect, two completions, and the reuse link — then reopen the ledger and confirm it again.
2. Resubmit the key with a changed instruction and watch the conflict name the dimension. Confirm zero new effects and find the refusal event. Then resubmit with the same operation but no authority and confirm a denial that reveals nothing.
3. Build a crash-after-effect adapter. Confirm FAILED with the effect present, retry with the same key, and write down exactly what the ledger does and does not establish about the world.
4. Race two threads on one key in a scratch directory. Count the effects. Write the number down next to the sequential result from step 1.

If you are building with an assistant:

Make repetition explicit: retry is a new attempt, replay returns a recorded
outcome, duplicate is the same effect twice. Bind each idempotency key to one
intended operation with a versioned fingerprint of capability, instruction,
payload, precondition, and performer; refuse collisions durably before any
effect. Check current authority before returning any recorded result, and let
denials reveal nothing about what is recorded. Do not re-evaluate the original
precondition on replay; do re-check it before fresh execution. Cache failures
as history, not as proof of no effect, and leave crash-gap reconciliation to
an explicit operator decision. Never claim exactly-once, transactions, or
concurrent safety without evidence.

Failure modes

• Minting fresh keys for old operations. A loop that creates new keys on every attempt manufactures duplicates, not safety.
• Treating the key as the operation. A lookup handle does not establish that two requests meant the same thing.
• Letting replay skip authority. A prior success is not permission for the current caller.
• Reading FAILED as “nothing happened.” A failure record is history, and the effect may already be real.
• Mistaking replayed failure for recovery. Returning the old FAILED did not fix anything; it only refused to make it worse.
• Re-checking preconditions on replay. Requiring the old world to still hold destroys the replay of the action that changed it.
• Assuming sequential safety covers concurrency. One key, two threads, two effects — the probe number goes here, not the hope.

What this chapter established

• Retry is a new attempt, replay is a returned record, duplicate is the same effect twice; the key alone establishes none of these.
• The historical demo shows safe duplicates, the key-only replay defect, fingerprint-bound call conflicts, cached crash-gap failures, and drift refusal — with the defect preserved as found, not rewritten. ⁸
• The Chapter 22 revision binds each key to a versioned operation fingerprint, checks current authority before any recorded result, refuses collisions durably with action.replay_refused, and returns history — including failures — labeled as history. ¹⁸
• Precondition identifies the instance but does not gate replay; a changed context needing a new decision needs a new key. ²
• Failed does not mean ineffectual. Failure caching now covers every adapter exception, not only RuntimeError; it still cannot cover a process death, whose orphaned request a same-key retry repeats. Actions have no reconcile path and concurrent duplicates are unprotected by design pending reconciliation first; the pinned rerun in experiments/applied-ai/evidence/retry-rerun/2026-09-14-1b3c7a2/ measures the frozen sequential semantics with an independent ledger-based verifier. ¹

Next

Repetition is now explicit: same requests return records, different requests conflict, unauthorized requests are denied, and unknown effects stay unknown. With acting, checking, and repeating all recorded, the process can entertain more than one proposal at a time — provided they cannot see each other.

Continue with Independent Calls.

References

• Collin Lee, Seo Jin Park, Ankita Kejriwal, Satoshi Matsushita, and John Ousterhout. Implementing Linearizability at Large Scale and Low Latency. SOSP, 2015. DOI.
• Andrew D. Birrell and Bruce Jay Nelson. Implementing Remote Procedure Calls. ACM Transactions on Computer Systems 2(1):39–59, 1984. Publication page.
• C. Mohan, Don Haderle, Bruce Lindsay, Hamid Pirahesh, and Peter Schwarz. ARIES: A Transaction Recovery Method Supporting Fine-Granularity Locking and Partial Rollbacks Using Write-Ahead Logging. ACM Transactions on Database Systems, 1992. Publication page.

Implementation sources: CodeAI baseline 7a0d43b for the historical demo; 1b3c7a2 (committed, containing the Chapters 19–22 revisions including the replay revision and the widened recorded-failure scope from RuntimeError to any adapter Exception), for current source, with the regression test tests/test_action_replay.py::test_non_runtime_error_after_effect_is_recorded_and_replayed_not_repeated. These are distinguished above. src/codeai/runtime.py: execute_action, _replay_action_result, _check_action_replay_fingerprint, _find_action_result, _action_fingerprint, ACTION_FINGERPRINT_V1, IdempotencyConflictError, invoke_recorded_call, _check_replay_fingerprint, _find_recorded_completion; src/codeai/adapters.py: ActionRequest, ActionResult; src/codeai/policy.py: PolicyEngine.require; src/codeai/ledger.py: SQLiteLedger.append; src/codeai/workstate.py: resume_call, NextOperation. Tests: tests/test_action_replay.py; updated tests/test_action_observation.py (recovery call repeats the original precondition); kept-green tests/test_runtime.py, tests/test_directive_registration.py, tests/test_check_binding.py. Teaching fragments were executed as illustrative code, not as pinned stage runs; the threaded probe is now a frozen diagnostic inside the pinned rerun. Evidence: experiments/applied-ai/evidence/retry-effects/ (README, producer, results, independent verifier), experiments/applied-ai/evidence/retry-rerun/2026-09-14-1b3c7a2/ (frozen protocol, producer, independent stdlib verifier, five seeded corruptions rejected), and experiments/applied-ai/evidence/working-state/2026-09-13-68f4ba0/. Footnotes mark provenance: source notes refer to inspected code, measurement notes to pinned runs, demo notes to preserved unpinned execution. Open future work is stated in prose, not footnotes. The retry-effects bundle is unchanged by this chapter.