# Advanced Agents From First Principles 39: How Do You Make an Agent Survive for Days? Build Durable Long-Running Workflows
# Book: Advanced Agents From First Principles
# Chapter URL: https://programmer.ie/books/advanced-agents-from-first-principles/39-chapter/
# Prose fingerprint (sha256, normalized): 11fa084ef93e0c181726d0aac377ad997af9ade5c0fddbb6830da7423701661d
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

An agent that works for thirty seconds can get away with a lot.
It can keep most of its state in memory.
It can assume the process will stay alive.
It can hold a Python object containing the plan.
It can retry a failed tool call directly.
It can wait synchronously for a result.
It can even use the conversation transcript as an approximate record of what has happened.
Then you ask the same system to work for six hours.
Or two days.
Or until a human approves something tomorrow morning.
Or until a deployment finishes.
Or until a customer replies.
Or until a cloud job emits an event.
Or until a retry window opens again.
Now the architecture changes.
The problem is no longer merely:
How should the model reason? It becomes:
How does the work remain correct while processes restart, workers disappear, providers fail, intent changes, state changes, humans go offline, and external systems finish whenever they finish? That is a workflow problem.
The core rule for this post is:
Make the workflow durable. Treat the model and worker as disposable.
This is the point where production agents stop looking like chat loops and start looking like distributed workflow systems with reasoning components inside them.
The Search Problem: “How Do I Build a Long-Running AI Agent?” A common first implementation looks like this:
async def run_agent(goal): plan = await model.plan(goal) for step in plan: result = await execute(step) if result.needs_human: await wait_for_human() if result.failed: await retry(step) return "done" This is understandable.
It is also fragile.
What happens if the process restarts while waiting for the human?
What happens if the worker dies after the external API accepted the request but before the result was persisted?
What happens if the workflow sleeps for three hours and the original deployment has been cancelled?
What happens if the approval arrives twice?
What happens if a timer fires after a superseding intent has replaced the original goal?
What happens if the model release changes while the workflow is paused?
What happens if a retry runs on a different worker in another region?
What happens if the task has an active commitment that must still be released after the business goal was cancelled?
The answer cannot be:
“Hopefully the LLM remembers.”
The model is not the durable state store.
The worker process is not the durable state store.
The prompt is not the durable state store.
The workflow runtime must be.
A Long-Running Agent Is Two Systems It helps to split the architecture explicitly.
DURABLE WORKFLOW PLANE intent versions goal state commitments waits timers retries approvals events checkpoints operation identities compensation / reconciliation │ ▼ REASONING / EXECUTION PLANE models search retrieval critics tools sandboxes workers verifiers The workflow plane answers:
What is supposed to happen next? What has already happened? What are we waiting for? What remains valid? What obligations still exist? The reasoning plane answers:
Given the current durable state, what candidate action or decision should we produce now? Those are different responsibilities.
The first should be deterministic whenever possible.
The second may be probabilistic.
That separation is fundamental.
The Workflow Must Survive the Worker Suppose an agent is performing a release:
1. inspect repository 2. propose migration 3. run tests 4. request approval 5. wait for approval 6. deploy 7. verify production If step 4 happens at 16:55 and the reviewer responds at 09:10 the next morning, the process that generated the request should not need to remain alive overnight.
The durable system should instead persist something like:
@dataclass(frozen=True) class WorkflowState: workflow_id: str workflow_version: int intent_id: str intent_version: int current_node: str status: str active_commitment_ids: tuple[str, ...] pending_waits: tuple[str, ...] completed_activity_ids: tuple[str, ...] checkpoint_ref: str | None behavioral_release_id: str policy_version: str Then the worker may disappear.
Tomorrow, another worker receives the approval event.
It loads the durable state.
It validates:
current intent; current state freshness; current authority; approval identity; behavioral-release compatibility; outstanding commitments; verifier requirements. Only then does it continue.
The continuation does not depend on process memory.
Durable Does Not Mean “Serialize the Conversation” A transcript is useful provenance.
It is not sufficient workflow state.
Consider:
Assistant: I am waiting for approval. What does that mean operationally?
Which approval?
For which candidate?
Bound to which artifact hash?
Who may approve?
When does it expire?
What intent version authorized the request?
What happens on timeout?
What commitment was created by asking the reviewer?
What state changes invalidate it?
The durable representation must make those facts explicit.
For example:
@dataclass(frozen=True) class ApprovalWait: wait_id: str workflow_id: str approval_request_id: str candidate_hash: str intent_id: str intent_version: int required_role: str expires_at: datetime timeout_policy: str state_version_vector: dict[str, str] Now “waiting for approval” is a real system state.
Workflow Nodes Should Be Semantic Do not make the durable workflow track every token generation or every internal reasoning step.
Persist semantic boundaries.
Useful workflow nodes include:
GATHER_EVIDENCE BUILD_CANDIDATE VERIFY_CANDIDATE REQUEST_APPROVAL WAIT_FOR_APPROVAL REVALIDATE COMMIT_OPERATION VERIFY_POSTCONDITION RECONCILE COMPLETE This aligns with the checkpoint principle from Step 35.
The durable system should know where it is in the business and control process.
It does not need to preserve every hidden computational detail of how the model arrived there.
Activities Are Disposable Attempts A durable workflow invokes activities.
Activities may execute on disposable workers.
For example:
async def inspect_repository_activity(ctx): ... async def run_tests_activity(ctx): ... async def generate_patch_activity(ctx): ... async def verify_patch_activity(ctx): ... async def deploy_activity(ctx): ... The durable workflow owns the lifecycle.
The activity owns one attempt.
That distinction is similar to the operation/attempt identity from Step 20.
logical activity │ ├── attempt 1 → timeout ├── attempt 2 → worker crash └── attempt 3 → success The workflow should not interpret three physical attempts as three logical operations.
Idempotency Is Mandatory at Durable Boundaries Long-running workflows are retried.
That is not optional.
Networks fail.
Workers crash.
Messages are delivered more than once.
Schedulers replay work.
Therefore an activity that creates side effects should carry a stable logical operation identity.
@dataclass(frozen=True) class ActivityCommand: workflow_id: str activity_id: str operation_id: str attempt_id: str intent_version: int fencing_epoch: int If the same logical operation is retried, the external mutation path must be able to recognize it.
same operation_id + new attempt_id is a retry.
It is not permission to create a second side effect.
Waiting Is a First-Class Workflow State A long-running agent spends enormous amounts of time not computing.
It waits for:
humans; webhooks; scheduled times; provider recovery; asynchronous jobs; approvals; CI; repository updates; market windows; business hours; rate-limit reset; external confirmation. Do not hold a worker while waiting.
Persist the wait.
RUNNING ↓ WAITING_FOR_EVENT ↓ worker released ↓ [event arrives] ↓ workflow reactivated This is a major operational difference between a durable workflow and a long-running coroutine.
Timers Must Also Be Durable Suppose the system says:
retry after 30 minutes A sleeping process is not a durable timer.
The timer belongs to workflow state.
@dataclass(frozen=True) class DurableTimer: timer_id: str workflow_id: str due_at: datetime purpose: str intent_version: int policy_version: str When it fires, that does not mean “perform the action.”
It means:
wake workflow ↓ reload authoritative state ↓ validate current intent ↓ validate current policy ↓ validate freshness ↓ continue if still valid A timer is a wake-up event.
It is not timeless authorization.
Delayed Work Must Revalidate Intent Step 37 made intent supersession explicit.
That becomes critical here.
Imagine:
10:00 schedule deployment retry for 11:00 10:30 user cancels deployment 11:00 timer fires The timer event must not resurrect the old objective.
The durable workflow checks:
if event.intent_version != intent_store.current_version(event.intent_id): return WorkflowDecision.SUPERSEDED The same applies to:
delayed queues; scheduled jobs; retries; human approvals; external callbacks; resumptions after outage. External Events Need Stable Identity Events may arrive twice.
They may arrive late.
They may arrive out of order.
They may refer to an old workflow generation.
A durable event should include stable correlation identity.
@dataclass(frozen=True) class WorkflowEvent: event_id: str workflow_id: str event_type: str external_ref: str | None observed_at: datetime payload_hash: str The workflow runtime should record consumed event IDs.
Duplicate delivery should be harmless.
Events Are Evidence, Not Commands This distinction matters.
Suppose GitHub emits:
CI completed successfully That event does not automatically mean:
deploy now The event updates workflow evidence.
The workflow then reevaluates:
current intent? current release? approval still valid? state still fresh? verifier sufficient? authority still granted? Then it decides what transition is allowed.
External events should not bypass the control plane.
Human Review Is Just Another Durable Wait — With Authority Human approval is special because it affects authority.
But operationally it is still a durable wait.
REQUEST_APPROVAL ↓ WAITING_FOR_APPROVAL ↓ reviewer response ↓ validate response ↓ revalidate candidate/state/intent ↓ continue or invalidate The model should not sit in a loop asking:
Has the human responded yet? The workflow should sleep until an authenticated event arrives.
A Human Response Must Be Bound to the Object Reviewed Step 29 already established this principle.
Durable workflows make it unavoidable.
Suppose the reviewer approves candidate hash:
sha256:abc123 While waiting, another process changes the candidate.
The eventual approval cannot authorize:
sha256:def456 The workflow needs an explicit transition:
APPROVAL_RECEIVED ↓ candidate hash mismatch ↓ APPROVAL_INVALIDATED ↓ REQUEST_NEW_APPROVAL Durability without identity binding would merely preserve stale authority more reliably.
Retries Belong to the Workflow Policy Do not bury retry behavior inside every tool implementation.
A workflow should make retry semantics observable.
@dataclass(frozen=True) class RetryPolicy: max_attempts: int initial_delay_seconds: int max_delay_seconds: int retryable_errors: tuple[str, ...] non_retryable_errors: tuple[str, ...] But advanced agents need more than generic exponential backoff.
A failure may mean:
transient infrastructure failure capacity unavailable state conflict intent superseded verification failure policy denial competence failure invalid candidate ambiguous external outcome Only some of these should retry.
A stale-state conflict should often trigger replan.
A policy denial should not retry.
An ambiguous side effect should reconcile before retry.
A verifier failure may require a different verifier path.
Retry Is Not Replanning This distinction matters.
retry = attempt the same logical operation again replan = choose a different strategy or action If a deployment API times out after accepting the deployment, blindly replanning may create a duplicate deployment.
If the implementation itself is invalid, retrying the same command is pointless.
The workflow needs to know which category it is dealing with.
Persist Retry History Retry policy should have memory outside the model.
@dataclass(frozen=True) class ActivityAttempt: activity_id: str attempt_number: int started_at: datetime finished_at: datetime | None outcome: str error_class: str | None external_operation_id: str | None This prevents:
worker restarts ↓ retry count forgotten ↓ retry forever It also supports incident forensics and SLO measurement.
Durable Workflow State Is Not Agent Memory These are different things.
Workflow state contains operational truth:
current node pending waits intent version commitment IDs completed activities operation IDs approval state retry counts Agent memory contains information that may help reasoning:
prior observations summaries preferences retrieved evidence learned heuristics Do not make memory the authoritative workflow state.
Memory may be stale.
Memory may be summarized incorrectly.
Memory may be unavailable on another placement.
The workflow must remain correct without trusting model memory.
Model Context Should Be Reconstructed When a workflow wakes after twelve hours, do not assume the original conversation context should simply be replayed wholesale.
Reconstruct a bounded context from durable state.
For example:
current intent current goal active commitments latest authoritative observations current workflow node relevant completed activities pending decision current policy constraints Then retrieve only the evidence needed for the next reasoning step.
This reduces context bloat and avoids carrying stale assumptions merely because they appeared earlier in the transcript.
A Workflow Can Outlive a Behavioral Release Suppose a workflow begins under:
release R17 It waits two days.
Production is now on:
release R18 What happens?
There are several options:
resume under R17 migrate workflow to R18 restart from a durable boundary require human review cancel / reconcile The correct answer depends on compatibility.
This is why Step 24’s behavioral release model and Step 35’s checkpoint compatibility matter.
The workflow must not silently wake under new behavior and pretend it is the same continuation.
Workflow Versioning Is Separate From Behavioral Release Versioning A workflow definition can change even if the model does not.
For example:
v1: verify → approve → deploy v2: verify → security scan → approve → deploy That is a workflow-schema change.
Long-running instances need migration policy.
@dataclass(frozen=True) class WorkflowDefinitionRef: workflow_name: str workflow_version: int Never assume all historical instances should interpret new code automatically.
Deterministic Workflow Definitions Are Valuable The runtime should prefer explicit transition logic.
class ReleaseWorkflow: def next(self, state, event): if state.status == "WAITING_FOR_APPROVAL": return self.handle_approval(state, event) if state.status == "WAITING_FOR_CI": return self.handle_ci(state, event) ... The model may help decide:
which candidate is best; how to repair a failure; what evidence to gather; which plan is promising. But the model does not need to decide whether an already-consumed approval event should be consumed again.
That is workflow logic.
Use the Model at Decision Points, Not as the Clock A healthy architecture looks like:
workflow reaches decision point ↓ assemble current context ↓ invoke model / search / critic ↓ produce structured candidate decision ↓ validate ↓ persist transition ↓ release worker Not:
model runs forever and remembers what time it is Commitments Must Survive Workflow Suspension Step 38 introduced durable commitments.
This becomes one of the most important reasons for durable workflow state.
Suppose a workflow reserves a deployment window and then waits six hours.
The commitment remains active even though no worker is running.
workflow suspended ≠ commitment suspended The commitment ledger should remain queryable by:
monitoring; cancellation logic; admission control; resource allocation; human operators; other agents. Workflow Completion Requires Commitment Closure A workflow should not report success while leaving unresolved commitments behind.
A completion invariant could be:
def can_complete(state, commitment_store): unresolved = commitment_store.unresolved_for_workflow(state.workflow_id) return not unresolved Or, more realistically, terminal state may allow known deferred obligations:
COMPLETE COMPLETE_WITH_DEFERRED_COMMITMENTS RECONCILIATION_REQUIRED The important thing is that unresolved obligations are explicit.
Cancellation Is a Workflow Transition Cancellation is not task.cancel().
The workflow should enter a durable cancellation path.
ACTIVE ↓ CANCELLATION_REQUESTED ↓ stop optional work ↓ fence future mutations ↓ inspect commitments ↓ reconcile in-flight operations ↓ release / compensate obligations ↓ CANCELLED This incorporates Step 37’s intent semantics.
A cancelled business objective may create more workflow work before the system is safely finished.
Cancellation Should Be Idempotent Users click twice.
Systems retry messages.
Operators issue the same cancellation repeatedly.
The transition should be stable.
cancel(cancelled_workflow) → still cancelled and must not create duplicate release or compensation operations.
Long-Running Workflows Need Protected Cleanup Capacity Imagine the system is overloaded.
The admission controller begins rejecting new speculative work.
That is reasonable.
But a workflow may still need to:
release a reservation; reconcile a payment; verify whether a deployment occurred; revoke a temporary capability; compensate an external mutation. Cleanup and reconciliation work should have protected capacity.
This follows the same logic as Step 16’s protected verification reserve.
optional exploration can be shed required reconciliation must retain a completion path Workflow Deadlines Are Different From Activity Timeouts Suppose an API activity has a 30-second timeout.
The business workflow may have a 24-hour deadline.
Those are different.
activity timeout = how long one execution attempt may run workflow deadline = when the overall obligation becomes unacceptable There may also be:
approval deadline commitment deadline freshness deadline external SLA deadline Treat them separately.
Deadline Expiry Is an Event, Not Automatic Failure When a deadline passes, the correct behavior depends on policy.
Possible transitions:
ESCALATE CANCEL RELEASE_COMMITMENT REPLAN DEGRADE HUMAN_REQUIRED BREACHED A model should not improvise this policy from prose.
Workflow State Should Be Append-Only Enough to Audit A simple mutable row may tell you the current state.
It may not tell you how you got there.
A production workflow benefits from a transition log:
@dataclass(frozen=True) class WorkflowTransition: transition_id: str workflow_id: str from_state: str to_state: str trigger_type: str trigger_ref: str intent_version: int occurred_at: datetime policy_version: str Then the current state can be reconstructed or checked against the event history.
This aligns naturally with Step 13 observability and Step 25 replay.
Event Sourcing Is Useful, but Not Mandatory Do not turn this into architecture fashion.
You may use:
an event-sourced workflow log; durable state rows plus append-only transitions; a workflow engine’s native persistence; a transactional state machine. The invariant matters more than the technology:
After process loss, the workflow can reconstruct what has happened and determine the next legal transition without relying on lost model memory.
Exactly-Once Execution Is Still Mostly a Mirage A durable workflow engine can give strong execution semantics internally.
It cannot magically make every external system exactly-once.
Suppose:
workflow → payment API The API may accept the payment and the response may be lost.
The durable workflow knows it attempted the call.
It still needs:
operation identity idempotency key external status query reconciliation Durability does not remove distributed ambiguity.
It makes ambiguity survivable.
Side Effects Should Live Behind Activities Keep workflow transition logic pure when possible.
workflow decision ↓ activity command ↓ side effect ↓ recorded result ↓ workflow transition This makes replay much safer.
Replaying workflow logic should not accidentally resend emails or redeploy production.
Replay the Workflow Without Replaying the World Step 25 distinguished recorded observation replay from live revalidation.
The same rule applies here.
A forensic replay should consume recorded activity outcomes:
activity A returned X approval event Y arrived commit operation Z produced result R It should not reissue the side effects merely because the workflow function is replayed.
This is one of the strongest reasons to separate orchestration from activities.
Workflow Determinism Has a Specific Meaning You do not need the model to be deterministic.
You need workflow history interpretation to be deterministic enough that replaying the same recorded transitions does not invent a different operational past.
Model calls should appear as recorded activity results or versioned decision artifacts.
For example:
MODEL_DECISION candidate_id = C42 release = R17 prompt_hash = ... output_hash = ... The workflow does not regenerate C42 during ordinary history replay.
A Model Retry Is a New Attempt, Not Historical Replay If you intentionally call the model again, that is new work.
Record it as such.
model attempt 1 → candidate C41 model attempt 2 → candidate C42 Do not silently overwrite the previous decision artifact.
This is important for trajectory observability and causal analysis.
Durable Workflows Need Explicit Wait Types Useful wait categories include:
WAIT_TIMER WAIT_HUMAN WAIT_EXTERNAL_JOB WAIT_RESOURCE WAIT_DEPENDENCY_HEALTH WAIT_EVENT WAIT_APPROVAL WAIT_RETRY_WINDOW WAIT_RECONCILIATION Each should have:
identity; creation reason; expiry/deadline; intent binding; relevant state versions; wake-up conditions. This is far safer than one generic sleeping=True flag.
Waiting Can Change Competence and Authority A workflow that was valid yesterday may not be valid today.
During a long wait:
the behavioral release may change; the verifier may become unavailable; the workload may move outside the competence envelope; the error budget may burn; policy may tighten; the authority class may change. Therefore wake-up logic should not simply continue from the next line.
It should pass through the relevant gates again.
Wake-Up Is a Revalidation Boundary A good general pattern is:
WAKE EVENT ↓ load durable workflow state ↓ validate intent ↓ validate workflow-version compatibility ↓ validate state freshness ↓ validate commitments ↓ validate competence ↓ validate authority ↓ validate placement ↓ continue Not every wake-up needs every expensive check.
Use dependency-aware invalidation from Step 36.
But conceptually, wake-up is a revalidation boundary.
The Workflow Should Know Why It Is Waiting Consider two workflows both paused for one hour.
One is waiting because:
provider rate limit The other because:
human approval required Those have completely different semantics.
The workflow state should say why the system is paused.
That affects:
alerting; SLO attribution; escalation; cancellation; budget accounting; authority. Long-Running Agents Need Liveness Invariants Correctness is not only about preventing bad actions.
A workflow can also fail by never finishing.
Useful invariants include:
no ACTIVE workflow without an owner or pending wait no WAITING workflow without a wake-up condition no retry path without a finite policy or escalation no reconciliation state without protected capacity no active commitment without a servicing path These are deterministic checks.
They do not require an LLM.
Detect Orphaned Workflows A workflow may become orphaned if:
the queue event was lost; a timer was never registered; a worker died before persisting the next transition; an external callback was mis-correlated; a release migration failed; a commitment still exists but the workflow says complete. Periodically scan durable workflow state for impossible or stale combinations.
For example:
def find_orphans(workflows, waits, commitments): ... Operational reconciliation is part of the architecture.
Workflow SLOs Are Not Model SLOs A model may respond successfully while the workflow is unhealthy.
Track things like:
workflow completion rate workflow age wait age stuck-workflow count retry amplification commitment breach rate reconciliation backlog human wait time wake-up latency stale-intent wakeups orphan rate These belong beside the behavioral reliability metrics from Step 27.
Measure Time by State For a long-running workflow, total latency alone is not very informative.
Break it down:
reasoning time external job time human wait time capacity wait time retry delay reconciliation time verification time Then you know what actually makes the workflow slow.
Do not optimize the model because humans take eight hours to approve the result.
Human Wait Time Is an Operational Dependency Step 33’s dependency graph already allowed human approval capacity as a dependency.
Long-running workflows make that measurable.
If critical workflows routinely block for six hours on one specialist reviewer, that is a real capacity constraint.
The architecture may need:
reviewer pools; delegation; escalation; better evidence packets; narrower approval scope; different authority policy. This is not primarily a model-quality problem.
Workflow Backpressure Matters If a dependency is unavailable, workflows accumulate.
Suppose a verifier outage causes 20,000 workflows to enter:
WAIT_VERIFIER When the verifier returns, waking all 20,000 immediately may recreate the outage.
Wake-up itself needs scheduling and backpressure.
dependency recovers ↓ controlled reactivation ↓ priority / deadlines / cohorts ↓ verifier protected from surge This connects directly to Steps 21 and 22.
Durable Workflow ≠ Workflow Engine Everywhere Not every agent needs this.
If your operation lasts 15 seconds and has no asynchronous dependencies or consequential side effects, a normal function may be enough.
Use the simplest mechanism that solves the failure.
You likely need durable workflow semantics when you have several of:
hours/days of elapsed time human waits external asynchronous jobs multiple side effects retries across process restarts commitments superseding intent distributed workers migration between placements strict audit/replay requirements Do not add a workflow engine because durable systems sound sophisticated.
A Minimal Durable Workflow Runtime The core can be surprisingly small conceptually.
from dataclasses import dataclass from enum import Enum class WorkflowStatus(str, Enum): RUNNABLE = "runnable" WAITING = "waiting" RECONCILING = "reconciling" COMPLETE = "complete" FAILED = "failed" CANCELLED = "cancelled" @dataclass(frozen=True) class WorkflowRecord: workflow_id: str workflow_version: int intent_id: str intent_version: int state_name: str status: WorkflowStatus fencing_epoch: int behavioral_release_id: str @dataclass(frozen=True) class TransitionRequest: workflow_id: str expected_workflow_version: int expected_intent_version: int from_state: str to_state: str trigger_id: str class WorkflowStore: def transition(self, request: TransitionRequest) -> WorkflowRecord: """Atomically compare expected state and append the next transition.""" raise NotImplementedError The model does not mutate WorkflowRecord directly.
It can propose structured decisions.
The durable runtime validates and persists transitions.
Separate Orchestrator Decisions From Model Decisions For example:
@dataclass(frozen=True) class CandidateDecision: candidate_id: str action_type: str evidence_refs: tuple[str, ...] The model may return CandidateDecision.
Then deterministic workflow logic checks:
is intent current? is candidate structurally valid? is required verification complete? is approval required? is authority sufficient? is state fresh? Only then does the workflow transition.
Do Not Let the Model Invent Workflow State Names Avoid designs where the model says:
{"next_state": "deployed_successfully"} and the runtime trusts it.
State transitions should come from a finite protocol.
The model may provide evidence relevant to a transition.
The control plane decides whether that transition is legal.
Model Calls Are Activities With Budgets Long-running workflows can accidentally spend enormous amounts of compute because each wake-up triggers fresh reasoning.
Track cumulative run budgets across the whole workflow.
model tokens model calls search expansions tool calls wall-clock active compute external API cost human-review requests The workflow persisting for three days must not reset its cost budget every morning.
This extends Step 16’s budget ledger into durable time.
Budget Resets Must Be Explicit Policy Sometimes a new intent version justifies a new budget.
Sometimes it should inherit the remaining budget.
Sometimes cancellation creates a separate reconciliation reserve.
Make that policy explicit.
Do not let restarts accidentally create free compute.
Durable Workflows Make “Agent Sessions” Less Important A session is a UI concept.
A workflow is an operational concept.
One workflow may span:
several conversations; several workers; several models; several days; several human reviewers. Conversely, one chat session may create multiple workflows.
Do not use chat-session identity as the primary operational identity.
Coding Agent Example Suppose an agent is implementing a large change.
workflow: upgrade authentication library Possible sequence:
inspect repository ↓ create isolated workspace ↓ generate migration candidate ↓ run tests ↓ WAIT_FOR_CI ↓ CI passes ↓ request human approval ↓ WAIT_FOR_APPROVAL ↓ reviewer requests change ↓ reopen reasoning activity ↓ new candidate ↓ verify ↓ approval ↓ revalidate main branch ↓ merge / commit ↓ postcondition verification This could take hours.
The model does not need to stay alive.
The workspace, evidence, intent, commitments and approvals do.
Research Agent Example A research workflow may wait for:
scheduled data release; document availability; human source approval; another specialist agent; market close; a benchmark result. The durable state can preserve:
research question claim graph source provenance pending evidence requests current confidence state verification gaps When new evidence arrives, only affected claims need reevaluation.
The whole investigation does not need to restart.
Browser Agent Example Suppose a browser agent is filling a complex government application.
It may need to:
collect documents wait for user upload resume form wait for external verification request confirmation submit verify submission ID Session state may expire while waiting.
The durable workflow should preserve the logical process while treating browser session state as refreshable/non-portable operational state.
The workflow can resume with:
new browser session fresh authoritative page state same logical workflow if compatibility permits.
DevOps Agent Example A deployment workflow is a natural durable workflow.
prepare release ↓ reserve window ↓ WAIT_WINDOW ↓ check current intent ↓ check dependency health ↓ deploy canary ↓ WAIT_OBSERVATION_WINDOW ↓ evaluate SLOs ↓ promote or rollback Trying to implement this as one LLM coroutine is unnecessary risk.
Durable Workflows Clarify Agent Boundaries An important consequence is that the agent becomes less mystical.
You can ask:
What state is this workflow in? What event is it waiting for? What commitments exist? Which transition is legal next? Which reasoning decision is actually needed? Instead of:
What is the agent thinking? That is a much better operational model.
Failure Mode: Process Memory as Workflow State Symptoms:
restart loses progress waits disappear retry counts reset cancellation forgotten Fix:
durable workflow state Failure Mode: Conversation Transcript as State Machine Symptoms:
model rereads history infers where it thinks the process is continues This is fragile because natural language is ambiguous.
Fix:
structured durable state + transcript as provenance Failure Mode: Sleeping Workers Symptoms:
thousands of processes sleeping while waiting for external events Fix:
persist wait + release worker Failure Mode: Retry Storm After Recovery Symptoms:
provider returns all waiting workflows wake provider fails again Fix:
backpressured reactivation Failure Mode: Lost Cancellation Symptoms:
worker never receives cancel signal later commits stale work Fix:
intent-version fencing at mutation boundary Failure Mode: Old Approval Resumes New Candidate Symptoms:
candidate changes while waiting old approval reused Fix:
approval bound to candidate/state/intent identity Failure Mode: Workflow Completes With Live Commitments Symptoms:
workflow says done reservation / external obligation remains active Fix:
terminal commitment invariants Failure Mode: New Release Silently Changes Paused Workflow Symptoms:
workflow sleeps under R17 wakes under R18 new semantics applied silently Fix:
behavioral-release compatibility gate Failure Mode: Replay Reissues Side Effects Symptoms:
forensic replay sends email again Fix:
activities recorded as historical results workflow replay does not rerun effects Failure Mode: Repeated Model Reasoning Burns Unlimited Budget Symptoms:
every wake-up triggers expensive full reasoning Fix:
durable cumulative budget ledger + decision-point-specific context Failure Mode: Everything Becomes a Workflow Symptoms:
simple 5-second task requires durable orchestration Fix:
use ordinary code when ordinary code is enough The series rule still applies:
Complexity must earn its place.
Failure Injection for Durable Agent Workflows Test the workflow runtime deliberately.
Worker dies before activity starts Expected:
activity safely retried Worker dies after external effect but before recording result Expected:
same operation identity external reconciliation no duplicate side effect Human approval arrives after supersession Expected:
approval rejected for stale intent Timer fires twice Expected:
idempotent wake-up Provider recovers after large backlog Expected:
controlled reactivation not thundering herd Behavioral release changes while workflow waits Expected:
compatibility decision before resume Commitment remains active when workflow tries to complete Expected:
completion blocked or explicitly marked deferred/reconciliation Workflow state write races with cancellation Expected:
optimistic concurrency rejects stale transition Duplicate external event arrives Expected:
same event ID ignored after first consumption Workflow sleeps past freshness budget Expected:
revalidation before next consequential step A Useful Workflow Transition Gate The important checks can be centralized.
class WorkflowTransitionGate: def validate(self, state, transition, context): if not context.intent_is_current(state.intent_id, state.intent_version): return "INTENT_SUPERSEDED" if not context.workflow_version_compatible(state): return "WORKFLOW_MIGRATION_REQUIRED" if transition.requires_fresh_state and not context.state_is_fresh(transition): return "REVALIDATION_REQUIRED" if transition.requires_authority and not context.authority_allows(transition): return "AUTHORITY_DENIED" if transition.requires_verification and not context.verification_valid(transition): return "VERIFICATION_REQUIRED" return "ALLOWED" Keep this deterministic.
The model can help produce candidates.
It should not decide whether stale intent is current.
Durable Workflow Outcomes Should Be Explicit Useful terminal or suspended outcomes include:
COMPLETE COMPLETE_WITH_DEFERRED_COMMITMENTS FAILED CANCELLED SUPERSEDED WAITING HUMAN_REQUIRED REVALIDATION_REQUIRED RECONCILIATION_REQUIRED WORKFLOW_MIGRATION_REQUIRED POLICY_BLOCKED Do not collapse all non-success states into generic failure.
Operationally, they mean very different things.
Benchmark Against Simpler Designs Before adopting a durable workflow engine, compare it against:
single process simple job queue job queue + durable cancellation coarse checkpoint/restart Measure:
recovery correctness side-effect duplication orphan rate operator burden implementation complexity latency cost If a simple queue solves the workload, use the simple queue.
The Deeper Architectural Shift We started this series thinking mostly about better reasoning.
Now consider where the model sits:
durable workflow │ ┌──────────┴──────────┐ │ │ deterministic reasoning state machine activity │ │ └──────────┬──────────┘ │ verified action The workflow survives:
model replacement; worker replacement; provider replacement; process restart; human delay; temporary outage. That is a much stronger architecture than asking one model invocation to somehow remain “the agent” for days.
The Principle A production long-running agent should not be defined by the lifetime of a process or conversation.
It should be defined by a durable operational identity.
That identity includes:
intent workflow state commitments external operation identities waits verification evidence history budgets policy Workers come and go.
Models come and go.
The workflow remains.
Make the workflow durable. Treat the model and worker as disposable.
That is how an agent survives for days without pretending that an LLM is a process manager, database, scheduler and distributed transaction coordinator.
And it leads directly to the next problem.
A durable workflow can survive partial failure.
But what happens when the system has already changed the world before a later step fails?
That is the subject of the next step:
transactions, compensation and reconciliation across external side effects.