How Do You Make an Agent Survive for Days? Build Durable Long-Running Workflows

An agent that works for thirty seconds can get away with a lot.

It can keep most of its state in memory.

It can assume the process will stay alive.

It can hold a Python object containing the plan.

It can retry a failed tool call directly.

It can wait synchronously for a result.

It can even use the conversation transcript as an approximate record of what has happened.

Then you ask the same system to work for six hours.

Or two days.

Or until a human approves something tomorrow morning.

Or until a deployment finishes.

Or until a customer replies.

Or until a cloud job emits an event.

Or until a retry window opens again.

Now the architecture changes.

The problem is no longer merely:

How should the model reason?

It becomes:

How does the work remain correct while
processes restart,
workers disappear,
providers fail,
intent changes,
state changes,
humans go offline,
and external systems finish whenever they finish?

That is a workflow problem.

The core rule for this post is:

Make the workflow durable. Treat the model and worker as disposable.

This is the point where production agents stop looking like chat loops and start looking like distributed workflow systems with reasoning components inside them.

The Search Problem: “How Do I Build a Long-Running AI Agent?”

A common first implementation looks like this:

async def run_agent(goal):
    plan = await model.plan(goal)

    for step in plan:
        result = await execute(step)

        if result.needs_human:
            await wait_for_human()

        if result.failed:
            await retry(step)

    return "done"

This is understandable.

It is also fragile.

What happens if the process restarts while waiting for the human?

What happens if the worker dies after the external API accepted the request but before the result was persisted?

What happens if the workflow sleeps for three hours and the original deployment has been cancelled?

What happens if the approval arrives twice?

What happens if a timer fires after a superseding intent has replaced the original goal?

What happens if the model release changes while the workflow is paused?

What happens if a retry runs on a different worker in another region?

What happens if the task has an active commitment that must still be released after the business goal was cancelled?

The answer cannot be:

“Hopefully the LLM remembers.”

The model is not the durable state store.

The worker process is not the durable state store.

The prompt is not the durable state store.

The workflow runtime must be.

A Long-Running Agent Is Two Systems

It helps to split the architecture explicitly.

                 DURABLE WORKFLOW PLANE

 intent versions
 goal state
 commitments
 waits
 timers
 retries
 approvals
 events
 checkpoints
 operation identities
 compensation / reconciliation

                         │
                         ▼

                 REASONING / EXECUTION PLANE

 models
 search
 retrieval
 critics
 tools
 sandboxes
 workers
 verifiers

The workflow plane answers:

What is supposed to happen next?
What has already happened?
What are we waiting for?
What remains valid?
What obligations still exist?

The reasoning plane answers:

Given the current durable state,
what candidate action or decision should we produce now?

Those are different responsibilities.

The first should be deterministic whenever possible.

The second may be probabilistic.

That separation is fundamental.

The Workflow Must Survive the Worker

Suppose an agent is performing a release:

1. inspect repository
2. propose migration
3. run tests
4. request approval
5. wait for approval
6. deploy
7. verify production

If step 4 happens at 16:55 and the reviewer responds at 09:10 the next morning, the process that generated the request should not need to remain alive overnight.

The durable system should instead persist something like:

@dataclass(frozen=True)
class WorkflowState:
    workflow_id: str
    workflow_version: int
    intent_id: str
    intent_version: int
    current_node: str
    status: str
    active_commitment_ids: tuple[str, ...]
    pending_waits: tuple[str, ...]
    completed_activity_ids: tuple[str, ...]
    checkpoint_ref: str | None
    behavioral_release_id: str
    policy_version: str

Then the worker may disappear.

Tomorrow, another worker receives the approval event.

It loads the durable state.

It validates:

• current intent;
• current state freshness;
• current authority;
• approval identity;
• behavioral-release compatibility;
• outstanding commitments;
• verifier requirements.

Only then does it continue.

The continuation does not depend on process memory.

Durable Does Not Mean “Serialize the Conversation”

A transcript is useful provenance.

It is not sufficient workflow state.

Consider:

Assistant: I am waiting for approval.

What does that mean operationally?

Which approval?

For which candidate?

Bound to which artifact hash?

Who may approve?

When does it expire?

What intent version authorized the request?

What happens on timeout?

What commitment was created by asking the reviewer?

What state changes invalidate it?

The durable representation must make those facts explicit.

For example:

@dataclass(frozen=True)
class ApprovalWait:
    wait_id: str
    workflow_id: str
    approval_request_id: str
    candidate_hash: str
    intent_id: str
    intent_version: int
    required_role: str
    expires_at: datetime
    timeout_policy: str
    state_version_vector: dict[str, str]

Now “waiting for approval” is a real system state.

Workflow Nodes Should Be Semantic

Do not make the durable workflow track every token generation or every internal reasoning step.

Persist semantic boundaries.

Useful workflow nodes include:

GATHER_EVIDENCE
BUILD_CANDIDATE
VERIFY_CANDIDATE
REQUEST_APPROVAL
WAIT_FOR_APPROVAL
REVALIDATE
COMMIT_OPERATION
VERIFY_POSTCONDITION
RECONCILE
COMPLETE

This aligns with the checkpoint principle from Step 35.

The durable system should know where it is in the business and control process.

It does not need to preserve every hidden computational detail of how the model arrived there.

Activities Are Disposable Attempts

A durable workflow invokes activities.

Activities may execute on disposable workers.

For example:

async def inspect_repository_activity(ctx): ...
async def run_tests_activity(ctx): ...
async def generate_patch_activity(ctx): ...
async def verify_patch_activity(ctx): ...
async def deploy_activity(ctx): ...

The durable workflow owns the lifecycle.

The activity owns one attempt.

That distinction is similar to the operation/attempt identity from Step 20.

logical activity
    │
    ├── attempt 1 → timeout
    ├── attempt 2 → worker crash
    └── attempt 3 → success

The workflow should not interpret three physical attempts as three logical operations.

Idempotency Is Mandatory at Durable Boundaries

Long-running workflows are retried.

That is not optional.

Networks fail.

Workers crash.

Messages are delivered more than once.

Schedulers replay work.

Therefore an activity that creates side effects should carry a stable logical operation identity.

@dataclass(frozen=True)
class ActivityCommand:
    workflow_id: str
    activity_id: str
    operation_id: str
    attempt_id: str
    intent_version: int
    fencing_epoch: int

If the same logical operation is retried, the external mutation path must be able to recognize it.

same operation_id
+ new attempt_id

is a retry.

It is not permission to create a second side effect.

Waiting Is a First-Class Workflow State

A long-running agent spends enormous amounts of time not computing.

It waits for:

• humans;
• webhooks;
• scheduled times;
• provider recovery;
• asynchronous jobs;
• approvals;
• CI;
• repository updates;
• market windows;
• business hours;
• rate-limit reset;
• external confirmation.

Do not hold a worker while waiting.

Persist the wait.

RUNNING
  ↓
WAITING_FOR_EVENT
  ↓
worker released
  ↓
[event arrives]
  ↓
workflow reactivated

This is a major operational difference between a durable workflow and a long-running coroutine.

Timers Must Also Be Durable

Suppose the system says:

retry after 30 minutes

A sleeping process is not a durable timer.

The timer belongs to workflow state.

@dataclass(frozen=True)
class DurableTimer:
    timer_id: str
    workflow_id: str
    due_at: datetime
    purpose: str
    intent_version: int
    policy_version: str

When it fires, that does not mean “perform the action.”

It means:

wake workflow
    ↓
reload authoritative state
    ↓
validate current intent
    ↓
validate current policy
    ↓
validate freshness
    ↓
continue if still valid

A timer is a wake-up event.

It is not timeless authorization.

Delayed Work Must Revalidate Intent

Step 37 made intent supersession explicit.

That becomes critical here.

Imagine:

10:00  schedule deployment retry for 11:00
10:30  user cancels deployment
11:00  timer fires

The timer event must not resurrect the old objective.

The durable workflow checks:

if event.intent_version != intent_store.current_version(event.intent_id):
    return WorkflowDecision.SUPERSEDED

The same applies to:

• delayed queues;
• scheduled jobs;
• retries;
• human approvals;
• external callbacks;
• resumptions after outage.

External Events Need Stable Identity

Events may arrive twice.

They may arrive late.

They may arrive out of order.

They may refer to an old workflow generation.

A durable event should include stable correlation identity.

@dataclass(frozen=True)
class WorkflowEvent:
    event_id: str
    workflow_id: str
    event_type: str
    external_ref: str | None
    observed_at: datetime
    payload_hash: str

The workflow runtime should record consumed event IDs.

Duplicate delivery should be harmless.

Events Are Evidence, Not Commands

This distinction matters.

Suppose GitHub emits:

CI completed successfully

That event does not automatically mean:

deploy now

The event updates workflow evidence.

The workflow then reevaluates:

current intent?
current release?
approval still valid?
state still fresh?
verifier sufficient?
authority still granted?

Then it decides what transition is allowed.

External events should not bypass the control plane.

Human Review Is Just Another Durable Wait — With Authority

Human approval is special because it affects authority.

But operationally it is still a durable wait.

REQUEST_APPROVAL
       ↓
WAITING_FOR_APPROVAL
       ↓
reviewer response
       ↓
validate response
       ↓
revalidate candidate/state/intent
       ↓
continue or invalidate

The model should not sit in a loop asking:

Has the human responded yet?

The workflow should sleep until an authenticated event arrives.

A Human Response Must Be Bound to the Object Reviewed

Step 29 already established this principle.

Durable workflows make it unavoidable.

Suppose the reviewer approves candidate hash:

sha256:abc123

While waiting, another process changes the candidate.

The eventual approval cannot authorize:

sha256:def456

The workflow needs an explicit transition:

APPROVAL_RECEIVED
        ↓
candidate hash mismatch
        ↓
APPROVAL_INVALIDATED
        ↓
REQUEST_NEW_APPROVAL

Durability without identity binding would merely preserve stale authority more reliably.

Retries Belong to the Workflow Policy

Do not bury retry behavior inside every tool implementation.

A workflow should make retry semantics observable.

@dataclass(frozen=True)
class RetryPolicy:
    max_attempts: int
    initial_delay_seconds: int
    max_delay_seconds: int
    retryable_errors: tuple[str, ...]
    non_retryable_errors: tuple[str, ...]

But advanced agents need more than generic exponential backoff.

A failure may mean:

transient infrastructure failure
capacity unavailable
state conflict
intent superseded
verification failure
policy denial
competence failure
invalid candidate
ambiguous external outcome

Only some of these should retry.

A stale-state conflict should often trigger replan.

A policy denial should not retry.

An ambiguous side effect should reconcile before retry.

A verifier failure may require a different verifier path.

Retry Is Not Replanning

This distinction matters.

retry
= attempt the same logical operation again

replan
= choose a different strategy or action

If a deployment API times out after accepting the deployment, blindly replanning may create a duplicate deployment.

If the implementation itself is invalid, retrying the same command is pointless.

The workflow needs to know which category it is dealing with.

Persist Retry History

Retry policy should have memory outside the model.

@dataclass(frozen=True)
class ActivityAttempt:
    activity_id: str
    attempt_number: int
    started_at: datetime
    finished_at: datetime | None
    outcome: str
    error_class: str | None
    external_operation_id: str | None

This prevents:

worker restarts
        ↓
retry count forgotten
        ↓
retry forever

It also supports incident forensics and SLO measurement.

Durable Workflow State Is Not Agent Memory

These are different things.

Workflow state contains operational truth:

current node
pending waits
intent version
commitment IDs
completed activities
operation IDs
approval state
retry counts

Agent memory contains information that may help reasoning:

prior observations
summaries
preferences
retrieved evidence
learned heuristics

Do not make memory the authoritative workflow state.

Memory may be stale.

Memory may be summarized incorrectly.

Memory may be unavailable on another placement.

The workflow must remain correct without trusting model memory.

Model Context Should Be Reconstructed

When a workflow wakes after twelve hours, do not assume the original conversation context should simply be replayed wholesale.

Reconstruct a bounded context from durable state.

For example:

current intent
current goal
active commitments
latest authoritative observations
current workflow node
relevant completed activities
pending decision
current policy constraints

Then retrieve only the evidence needed for the next reasoning step.

This reduces context bloat and avoids carrying stale assumptions merely because they appeared earlier in the transcript.

A Workflow Can Outlive a Behavioral Release

Suppose a workflow begins under:

release R17

It waits two days.

Production is now on:

release R18

What happens?

There are several options:

resume under R17
migrate workflow to R18
restart from a durable boundary
require human review
cancel / reconcile

The correct answer depends on compatibility.

This is why Step 24’s behavioral release model and Step 35’s checkpoint compatibility matter.

The workflow must not silently wake under new behavior and pretend it is the same continuation.

Workflow Versioning Is Separate From Behavioral Release Versioning

A workflow definition can change even if the model does not.

For example:

v1:
verify → approve → deploy

v2:
verify → security scan → approve → deploy

That is a workflow-schema change.

Long-running instances need migration policy.

@dataclass(frozen=True)
class WorkflowDefinitionRef:
    workflow_name: str
    workflow_version: int

Never assume all historical instances should interpret new code automatically.

Deterministic Workflow Definitions Are Valuable

The runtime should prefer explicit transition logic.

class ReleaseWorkflow:
    def next(self, state, event):
        if state.status == "WAITING_FOR_APPROVAL":
            return self.handle_approval(state, event)

        if state.status == "WAITING_FOR_CI":
            return self.handle_ci(state, event)

        ...

The model may help decide:

• which candidate is best;
• how to repair a failure;
• what evidence to gather;
• which plan is promising.

But the model does not need to decide whether an already-consumed approval event should be consumed again.

That is workflow logic.

Use the Model at Decision Points, Not as the Clock

A healthy architecture looks like:

workflow reaches decision point
        ↓
assemble current context
        ↓
invoke model / search / critic
        ↓
produce structured candidate decision
        ↓
validate
        ↓
persist transition
        ↓
release worker

Not:

model runs forever
and remembers what time it is

Commitments Must Survive Workflow Suspension

Step 38 introduced durable commitments.

This becomes one of the most important reasons for durable workflow state.

Suppose a workflow reserves a deployment window and then waits six hours.

The commitment remains active even though no worker is running.

workflow suspended
≠
commitment suspended

The commitment ledger should remain queryable by:

• monitoring;
• cancellation logic;
• admission control;
• resource allocation;
• human operators;
• other agents.

Workflow Completion Requires Commitment Closure

A workflow should not report success while leaving unresolved commitments behind.

A completion invariant could be:

def can_complete(state, commitment_store):
    unresolved = commitment_store.unresolved_for_workflow(state.workflow_id)
    return not unresolved

Or, more realistically, terminal state may allow known deferred obligations:

COMPLETE
COMPLETE_WITH_DEFERRED_COMMITMENTS
RECONCILIATION_REQUIRED

The important thing is that unresolved obligations are explicit.

Cancellation Is a Workflow Transition

Cancellation is not task.cancel().

The workflow should enter a durable cancellation path.

ACTIVE
  ↓
CANCELLATION_REQUESTED
  ↓
stop optional work
  ↓
fence future mutations
  ↓
inspect commitments
  ↓
reconcile in-flight operations
  ↓
release / compensate obligations
  ↓
CANCELLED

This incorporates Step 37’s intent semantics.

A cancelled business objective may create more workflow work before the system is safely finished.

Cancellation Should Be Idempotent

Users click twice.

Systems retry messages.

Operators issue the same cancellation repeatedly.

The transition should be stable.

cancel(cancelled_workflow)
    → still cancelled

and must not create duplicate release or compensation operations.

Long-Running Workflows Need Protected Cleanup Capacity

Imagine the system is overloaded.

The admission controller begins rejecting new speculative work.

That is reasonable.

But a workflow may still need to:

• release a reservation;
• reconcile a payment;
• verify whether a deployment occurred;
• revoke a temporary capability;
• compensate an external mutation.

Cleanup and reconciliation work should have protected capacity.

This follows the same logic as Step 16’s protected verification reserve.

optional exploration
can be shed

required reconciliation
must retain a completion path

Workflow Deadlines Are Different From Activity Timeouts

Suppose an API activity has a 30-second timeout.

The business workflow may have a 24-hour deadline.

Those are different.

activity timeout
= how long one execution attempt may run

workflow deadline
= when the overall obligation becomes unacceptable

There may also be:

approval deadline
commitment deadline
freshness deadline
external SLA deadline

Treat them separately.

Deadline Expiry Is an Event, Not Automatic Failure

When a deadline passes, the correct behavior depends on policy.

Possible transitions:

ESCALATE
CANCEL
RELEASE_COMMITMENT
REPLAN
DEGRADE
HUMAN_REQUIRED
BREACHED

A model should not improvise this policy from prose.

Workflow State Should Be Append-Only Enough to Audit

A simple mutable row may tell you the current state.

It may not tell you how you got there.

A production workflow benefits from a transition log:

@dataclass(frozen=True)
class WorkflowTransition:
    transition_id: str
    workflow_id: str
    from_state: str
    to_state: str
    trigger_type: str
    trigger_ref: str
    intent_version: int
    occurred_at: datetime
    policy_version: str

Then the current state can be reconstructed or checked against the event history.

This aligns naturally with Step 13 observability and Step 25 replay.

Event Sourcing Is Useful, but Not Mandatory

Do not turn this into architecture fashion.

You may use:

• an event-sourced workflow log;
• durable state rows plus append-only transitions;
• a workflow engine’s native persistence;
• a transactional state machine.

The invariant matters more than the technology:

After process loss, the workflow can reconstruct what has happened and determine the next legal transition without relying on lost model memory.

Exactly-Once Execution Is Still Mostly a Mirage

A durable workflow engine can give strong execution semantics internally.

It cannot magically make every external system exactly-once.

Suppose:

workflow → payment API

The API may accept the payment and the response may be lost.

The durable workflow knows it attempted the call.

It still needs:

operation identity
idempotency key
external status query
reconciliation

Durability does not remove distributed ambiguity.

It makes ambiguity survivable.

Side Effects Should Live Behind Activities

Keep workflow transition logic pure when possible.

workflow decision
      ↓
activity command
      ↓
side effect
      ↓
recorded result
      ↓
workflow transition

This makes replay much safer.

Replaying workflow logic should not accidentally resend emails or redeploy production.

Replay the Workflow Without Replaying the World

Step 25 distinguished recorded observation replay from live revalidation.

The same rule applies here.

A forensic replay should consume recorded activity outcomes:

activity A returned X
approval event Y arrived
commit operation Z produced result R

It should not reissue the side effects merely because the workflow function is replayed.

This is one of the strongest reasons to separate orchestration from activities.

Workflow Determinism Has a Specific Meaning

You do not need the model to be deterministic.

You need workflow history interpretation to be deterministic enough that replaying the same recorded transitions does not invent a different operational past.

Model calls should appear as recorded activity results or versioned decision artifacts.

For example:

MODEL_DECISION
candidate_id = C42
release = R17
prompt_hash = ...
output_hash = ...

The workflow does not regenerate C42 during ordinary history replay.

A Model Retry Is a New Attempt, Not Historical Replay

If you intentionally call the model again, that is new work.

Record it as such.

model attempt 1 → candidate C41
model attempt 2 → candidate C42

Do not silently overwrite the previous decision artifact.

This is important for trajectory observability and causal analysis.

Durable Workflows Need Explicit Wait Types

Useful wait categories include:

WAIT_TIMER
WAIT_HUMAN
WAIT_EXTERNAL_JOB
WAIT_RESOURCE
WAIT_DEPENDENCY_HEALTH
WAIT_EVENT
WAIT_APPROVAL
WAIT_RETRY_WINDOW
WAIT_RECONCILIATION

Each should have:

• identity;
• creation reason;
• expiry/deadline;
• intent binding;
• relevant state versions;
• wake-up conditions.

This is far safer than one generic sleeping=True flag.

Waiting Can Change Competence and Authority

A workflow that was valid yesterday may not be valid today.

During a long wait:

• the behavioral release may change;
• the verifier may become unavailable;
• the workload may move outside the competence envelope;
• the error budget may burn;
• policy may tighten;
• the authority class may change.

Therefore wake-up logic should not simply continue from the next line.

It should pass through the relevant gates again.

Wake-Up Is a Revalidation Boundary

A good general pattern is:

WAKE EVENT
    ↓
load durable workflow state
    ↓
validate intent
    ↓
validate workflow-version compatibility
    ↓
validate state freshness
    ↓
validate commitments
    ↓
validate competence
    ↓
validate authority
    ↓
validate placement
    ↓
continue

Not every wake-up needs every expensive check.

Use dependency-aware invalidation from Step 36.

But conceptually, wake-up is a revalidation boundary.

The Workflow Should Know Why It Is Waiting

Consider two workflows both paused for one hour.

One is waiting because:

provider rate limit

The other because:

human approval required

Those have completely different semantics.

The workflow state should say why the system is paused.

That affects:

• alerting;
• SLO attribution;
• escalation;
• cancellation;
• budget accounting;
• authority.

Long-Running Agents Need Liveness Invariants

Correctness is not only about preventing bad actions.

A workflow can also fail by never finishing.

Useful invariants include:

no ACTIVE workflow without an owner or pending wait
no WAITING workflow without a wake-up condition
no retry path without a finite policy or escalation
no reconciliation state without protected capacity
no active commitment without a servicing path

These are deterministic checks.

They do not require an LLM.

Detect Orphaned Workflows

A workflow may become orphaned if:

• the queue event was lost;
• a timer was never registered;
• a worker died before persisting the next transition;
• an external callback was mis-correlated;
• a release migration failed;
• a commitment still exists but the workflow says complete.

Periodically scan durable workflow state for impossible or stale combinations.

For example:

def find_orphans(workflows, waits, commitments):
    ...

Operational reconciliation is part of the architecture.

Workflow SLOs Are Not Model SLOs

A model may respond successfully while the workflow is unhealthy.

Track things like:

workflow completion rate
workflow age
wait age
stuck-workflow count
retry amplification
commitment breach rate
reconciliation backlog
human wait time
wake-up latency
stale-intent wakeups
orphan rate

These belong beside the behavioral reliability metrics from Step 27.

Measure Time by State

For a long-running workflow, total latency alone is not very informative.

Break it down:

reasoning time
external job time
human wait time
capacity wait time
retry delay
reconciliation time
verification time

Then you know what actually makes the workflow slow.

Do not optimize the model because humans take eight hours to approve the result.

Human Wait Time Is an Operational Dependency

Step 33’s dependency graph already allowed human approval capacity as a dependency.

Long-running workflows make that measurable.

If critical workflows routinely block for six hours on one specialist reviewer, that is a real capacity constraint.

The architecture may need:

• reviewer pools;
• delegation;
• escalation;
• better evidence packets;
• narrower approval scope;
• different authority policy.

This is not primarily a model-quality problem.

Workflow Backpressure Matters

If a dependency is unavailable, workflows accumulate.

Suppose a verifier outage causes 20,000 workflows to enter:

WAIT_VERIFIER

When the verifier returns, waking all 20,000 immediately may recreate the outage.

Wake-up itself needs scheduling and backpressure.

dependency recovers
      ↓
controlled reactivation
      ↓
priority / deadlines / cohorts
      ↓
verifier protected from surge

This connects directly to Steps 21 and 22.

Durable Workflow ≠ Workflow Engine Everywhere

Not every agent needs this.

If your operation lasts 15 seconds and has no asynchronous dependencies or consequential side effects, a normal function may be enough.

Use the simplest mechanism that solves the failure.

You likely need durable workflow semantics when you have several of:

hours/days of elapsed time
human waits
external asynchronous jobs
multiple side effects
retries across process restarts
commitments
superseding intent
distributed workers
migration between placements
strict audit/replay requirements

Do not add a workflow engine because durable systems sound sophisticated.

A Minimal Durable Workflow Runtime

The core can be surprisingly small conceptually.

from dataclasses import dataclass
from enum import Enum


class WorkflowStatus(str, Enum):
    RUNNABLE = "runnable"
    WAITING = "waiting"
    RECONCILING = "reconciling"
    COMPLETE = "complete"
    FAILED = "failed"
    CANCELLED = "cancelled"


@dataclass(frozen=True)
class WorkflowRecord:
    workflow_id: str
    workflow_version: int
    intent_id: str
    intent_version: int
    state_name: str
    status: WorkflowStatus
    fencing_epoch: int
    behavioral_release_id: str


@dataclass(frozen=True)
class TransitionRequest:
    workflow_id: str
    expected_workflow_version: int
    expected_intent_version: int
    from_state: str
    to_state: str
    trigger_id: str


class WorkflowStore:
    def transition(self, request: TransitionRequest) -> WorkflowRecord:
        """Atomically compare expected state and append the next transition."""
        raise NotImplementedError

The model does not mutate WorkflowRecord directly.

It can propose structured decisions.

The durable runtime validates and persists transitions.

Separate Orchestrator Decisions From Model Decisions

For example:

@dataclass(frozen=True)
class CandidateDecision:
    candidate_id: str
    action_type: str
    evidence_refs: tuple[str, ...]

The model may return CandidateDecision.

Then deterministic workflow logic checks:

is intent current?
is candidate structurally valid?
is required verification complete?
is approval required?
is authority sufficient?
is state fresh?

Only then does the workflow transition.

Do Not Let the Model Invent Workflow State Names

Avoid designs where the model says:

{"next_state": "deployed_successfully"}

and the runtime trusts it.

State transitions should come from a finite protocol.

The model may provide evidence relevant to a transition.

The control plane decides whether that transition is legal.

Model Calls Are Activities With Budgets

Long-running workflows can accidentally spend enormous amounts of compute because each wake-up triggers fresh reasoning.

Track cumulative run budgets across the whole workflow.

model tokens
model calls
search expansions
tool calls
wall-clock active compute
external API cost
human-review requests

The workflow persisting for three days must not reset its cost budget every morning.

This extends Step 16’s budget ledger into durable time.

Budget Resets Must Be Explicit Policy

Sometimes a new intent version justifies a new budget.

Sometimes it should inherit the remaining budget.

Sometimes cancellation creates a separate reconciliation reserve.

Make that policy explicit.

Do not let restarts accidentally create free compute.

Durable Workflows Make “Agent Sessions” Less Important

A session is a UI concept.

A workflow is an operational concept.

One workflow may span:

• several conversations;
• several workers;
• several models;
• several days;
• several human reviewers.

Conversely, one chat session may create multiple workflows.

Do not use chat-session identity as the primary operational identity.

Coding Agent Example

Suppose an agent is implementing a large change.

workflow: upgrade authentication library

Possible sequence:

inspect repository
      ↓
create isolated workspace
      ↓
generate migration candidate
      ↓
run tests
      ↓
WAIT_FOR_CI
      ↓
CI passes
      ↓
request human approval
      ↓
WAIT_FOR_APPROVAL
      ↓
reviewer requests change
      ↓
reopen reasoning activity
      ↓
new candidate
      ↓
verify
      ↓
approval
      ↓
revalidate main branch
      ↓
merge / commit
      ↓
postcondition verification

This could take hours.

The model does not need to stay alive.

The workspace, evidence, intent, commitments and approvals do.

Research Agent Example

A research workflow may wait for:

• scheduled data release;
• document availability;
• human source approval;
• another specialist agent;
• market close;
• a benchmark result.

The durable state can preserve:

research question
claim graph
source provenance
pending evidence requests
current confidence state
verification gaps

When new evidence arrives, only affected claims need reevaluation.

The whole investigation does not need to restart.

Browser Agent Example

Suppose a browser agent is filling a complex government application.

It may need to:

collect documents
wait for user upload
resume form
wait for external verification
request confirmation
submit
verify submission ID

Session state may expire while waiting.

The durable workflow should preserve the logical process while treating browser session state as refreshable/non-portable operational state.

The workflow can resume with:

new browser session
fresh authoritative page state
same logical workflow

if compatibility permits.

DevOps Agent Example

A deployment workflow is a natural durable workflow.

prepare release
      ↓
reserve window
      ↓
WAIT_WINDOW
      ↓
check current intent
      ↓
check dependency health
      ↓
deploy canary
      ↓
WAIT_OBSERVATION_WINDOW
      ↓
evaluate SLOs
      ↓
promote or rollback

Trying to implement this as one LLM coroutine is unnecessary risk.

Durable Workflows Clarify Agent Boundaries

An important consequence is that the agent becomes less mystical.

You can ask:

What state is this workflow in?
What event is it waiting for?
What commitments exist?
Which transition is legal next?
Which reasoning decision is actually needed?

Instead of:

What is the agent thinking?

That is a much better operational model.

Failure Mode: Process Memory as Workflow State

Symptoms:

restart loses progress
waits disappear
retry counts reset
cancellation forgotten

Fix:

durable workflow state

Failure Mode: Conversation Transcript as State Machine

Symptoms:

model rereads history
infers where it thinks the process is
continues

This is fragile because natural language is ambiguous.

Fix:

structured durable state + transcript as provenance

Failure Mode: Sleeping Workers

Symptoms:

thousands of processes sleeping
while waiting for external events

Fix:

persist wait + release worker

Failure Mode: Retry Storm After Recovery

Symptoms:

provider returns
all waiting workflows wake
provider fails again

Fix:

backpressured reactivation

Failure Mode: Lost Cancellation

Symptoms:

worker never receives cancel signal
later commits stale work

Fix:

intent-version fencing at mutation boundary

Failure Mode: Old Approval Resumes New Candidate

Symptoms:

candidate changes while waiting
old approval reused

Fix:

approval bound to candidate/state/intent identity

Failure Mode: Workflow Completes With Live Commitments

Symptoms:

workflow says done
reservation / external obligation remains active

Fix:

terminal commitment invariants

Failure Mode: New Release Silently Changes Paused Workflow

Symptoms:

workflow sleeps under R17
wakes under R18
new semantics applied silently

Fix:

behavioral-release compatibility gate

Failure Mode: Replay Reissues Side Effects

Symptoms:

forensic replay sends email again

Fix:

activities recorded as historical results
workflow replay does not rerun effects

Failure Mode: Repeated Model Reasoning Burns Unlimited Budget

Symptoms:

every wake-up triggers expensive full reasoning

Fix:

durable cumulative budget ledger
+ decision-point-specific context

Failure Mode: Everything Becomes a Workflow

Symptoms:

simple 5-second task requires durable orchestration

Fix:

use ordinary code when ordinary code is enough

The series rule still applies:

Complexity must earn its place.

Failure Injection for Durable Agent Workflows

Test the workflow runtime deliberately.

Worker dies before activity starts

Expected:

activity safely retried

Worker dies after external effect but before recording result

Expected:

same operation identity
external reconciliation
no duplicate side effect

Human approval arrives after supersession

Expected:

approval rejected for stale intent

Timer fires twice

Expected:

idempotent wake-up

Provider recovers after large backlog

Expected:

controlled reactivation
not thundering herd

Behavioral release changes while workflow waits

Expected:

compatibility decision before resume

Commitment remains active when workflow tries to complete

Expected:

completion blocked or explicitly marked deferred/reconciliation

Workflow state write races with cancellation

Expected:

optimistic concurrency rejects stale transition

Duplicate external event arrives

Expected:

same event ID ignored after first consumption

Workflow sleeps past freshness budget

Expected:

revalidation before next consequential step

A Useful Workflow Transition Gate

The important checks can be centralized.

class WorkflowTransitionGate:
    def validate(self, state, transition, context):
        if not context.intent_is_current(state.intent_id, state.intent_version):
            return "INTENT_SUPERSEDED"

        if not context.workflow_version_compatible(state):
            return "WORKFLOW_MIGRATION_REQUIRED"

        if transition.requires_fresh_state and not context.state_is_fresh(transition):
            return "REVALIDATION_REQUIRED"

        if transition.requires_authority and not context.authority_allows(transition):
            return "AUTHORITY_DENIED"

        if transition.requires_verification and not context.verification_valid(transition):
            return "VERIFICATION_REQUIRED"

        return "ALLOWED"

Keep this deterministic.

The model can help produce candidates.

It should not decide whether stale intent is current.

Durable Workflow Outcomes Should Be Explicit

Useful terminal or suspended outcomes include:

COMPLETE
COMPLETE_WITH_DEFERRED_COMMITMENTS
FAILED
CANCELLED
SUPERSEDED
WAITING
HUMAN_REQUIRED
REVALIDATION_REQUIRED
RECONCILIATION_REQUIRED
WORKFLOW_MIGRATION_REQUIRED
POLICY_BLOCKED

Do not collapse all non-success states into generic failure.

Operationally, they mean very different things.

Benchmark Against Simpler Designs

Before adopting a durable workflow engine, compare it against:

single process
simple job queue
job queue + durable cancellation
coarse checkpoint/restart

Measure:

recovery correctness
side-effect duplication
orphan rate
operator burden
implementation complexity
latency
cost

If a simple queue solves the workload, use the simple queue.

The Deeper Architectural Shift

We started this series thinking mostly about better reasoning.

Now consider where the model sits:

                 durable workflow
                       │
            ┌──────────┴──────────┐
            │                     │
       deterministic          reasoning
       state machine            activity
            │                     │
            └──────────┬──────────┘
                       │
                  verified action

The workflow survives:

• model replacement;
• worker replacement;
• provider replacement;
• process restart;
• human delay;
• temporary outage.

That is a much stronger architecture than asking one model invocation to somehow remain “the agent” for days.

The Principle

A production long-running agent should not be defined by the lifetime of a process or conversation.

It should be defined by a durable operational identity.

That identity includes:

intent
workflow state
commitments
external operation identities
waits
verification evidence
history
budgets
policy

Workers come and go.

Models come and go.

The workflow remains.

Make the workflow durable. Treat the model and worker as disposable.

That is how an agent survives for days without pretending that an LLM is a process manager, database, scheduler and distributed transaction coordinator.

And it leads directly to the next problem.

A durable workflow can survive partial failure.

But what happens when the system has already changed the world before a later step fails?

That is the subject of the next step:

transactions, compensation and reconciliation across external side effects.