# Advanced Agents From First Principles 35: How Do You Move a Running Agent Between Workers Without Losing Meaning? Build Portable Execution State and Safe Handoff
# Book: Advanced Agents From First Principles
# Chapter URL: https://programmer.ie/books/advanced-agents-from-first-principles/35-chapter/
# Prose fingerprint (sha256, normalized): c47843f4656959e6eb816532179a0f6b62df5f829d8e9e504ae01366842ae6d9
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

A long-running agent starts on one worker.
Then reality intervenes.
The GPU is reclaimed.
A provider becomes unhealthy.
A region loses capacity.
The browser pool is drained.
A more suitable model becomes available.
The scheduler decides the run should move.
So you checkpoint the agent and resume it somewhere else.
Simple.
Except it is not.
An agent run is not merely a Python object containing a few messages.
By the time a serious run has been executing for several minutes, it may have accumulated:
a task interpretation, retrieved evidence, tool observations, working memory, search branches, rejected candidates, budget consumption, verifier evidence, external side effects, pending operations, approval state, lease ownership, fencing epochs, placement assumptions, capability and authority decisions, environment identity, release identity, and provenance links connecting all of those things. If we move only the conversation transcript, we have not moved the run.
We have created a new run that merely remembers some text from the old one.
That distinction matters.
Step 34 gave us capability-aware placement across heterogeneous execution targets.
Now we need to answer the next question:
How can a running agent move between workers, models, providers or regions without silently changing the meaning, evidence, ownership or authority of the run?
The core rule for this post is:
A task may move. Its meaning, evidence, and authority must not move implicitly.
The Search Problem: “How Do I Resume an AI Agent on Another Worker Safely?” A naive checkpoint often looks like this:
checkpoint = { "messages": messages, "current_step": step, } That may be enough for a toy chatbot.
It is not enough for a production agent that can observe changing external state or perform side effects.
Suppose a coding agent has:
read repository commit A, generated patch P, run tests against workspace W, received verifier evidence V, requested approval for the exact hash of P, then loses its worker before commit. The replacement worker must not simply deserialize the patch and continue.
It needs to know:
Was P verified against the same repository state? Is approval still valid for this exact artifact? Has another worker already committed the operation? Has the repository changed since verification? Does the replacement worker have the same authority? Is its tool contract compatible? Is the verifier evidence still applicable? Without those answers, “resume” can become duplicated mutation, stale execution, authority escalation or verifier misbinding.
Portable execution state is therefore not serialization.
It is a continuation contract.
1. Separate Logical State From Physical Worker State The first architectural distinction is:
logical run state ≠ worker process state A worker may contain:
open sockets, Python objects, GPU tensors, browser handles, temporary directories, model KV caches, file descriptors, local locks, process IDs. Most of those are not portable.
The logical run should instead be reconstructable from durable references.
For example:
from dataclasses import dataclass @dataclass(frozen=True) class ExecutionCheckpoint: run_id: str checkpoint_id: str release_id: str state_schema_version: str task_state_ref: str observation_manifest_ref: str workspace_snapshot_ref: str | None memory_snapshot_ref: str | None search_state_ref: str | None verifier_evidence_ref: str | None operation_ledger_ref: str authority_context_ref: str budget_ledger_ref: str placement_ref: str owner_epoch: int checkpoint_hash: str The checkpoint should point to durable, versioned artifacts rather than assume the replacement worker can reconstruct hidden process memory.
2. Define Resumability Classes Not every step is equally resumable.
A useful runtime can classify execution points explicitly.
R0 — not resumable R1 — restart from previous durable boundary R2 — resume from deterministic state R3 — resume with recorded observations R4 — resume after ownership transfer R5 — resume after external side effects with reconciliation This classification prevents the scheduler from assuming every interruption can be solved by loading a checkpoint.
Example A pure planning step might be R2.
A research step using recorded web observations might be R3.
A coding task in an isolated workspace may be R4.
A task that sent an external API request and timed out before receiving confirmation may require R5 reconciliation.
Sometimes the correct state is simply:
MIGRATION_UNSAFE That is better than pretending portability exists where it does not.
3. Checkpoint at Semantic Boundaries Do not checkpoint at arbitrary process locations if the state cannot be interpreted safely.
Prefer semantic boundaries such as:
observation completed candidate produced candidate verified approval received operation prepared operation committed postcondition verified Those boundaries correspond to meaningful state transitions.
For example:
GENERATING ↓ CANDIDATE_READY ↓ VERIFYING ↓ VERIFIED ↓ AWAITING_AUTHORITY ↓ AUTHORIZED ↓ COMMITTING ↓ COMMITTED ↓ POSTCONDITION_VERIFIED A migration in CANDIDATE_READY is usually easier than one halfway through COMMITTING.
This is the same reason databases care about transaction boundaries.
Agent runtimes should too.
4. Make Ownership Transfer Explicit Step 20 introduced leases and fencing for distributed work.
Handoff must use them.
The dangerous pattern is:
worker A appears dead worker B resumes worker A wakes up both continue If both workers can mutate external state, the checkpoint system has created a split-brain agent.
A safe transfer looks more like:
worker A owns epoch 41 ↓ checkpoint sealed ↓ lease revoked / expires ↓ coordinator advances epoch to 42 ↓ worker B receives epoch 42 ↓ mutation gateway rejects epoch 41 The important rule is:
Checkpoint possession does not imply mutation authority.
Only the current fencing epoch should authorize consequential side effects.
5. Never Transfer Authority by Copying Credentials A portable checkpoint should not contain broad persistent credentials that magically grant the new worker the same privileges.
Instead, authority should be reconstructed from policy.
For example:
checkpoint says: required authority = A3 approved operation = op-847 approved artifact hash = sha256:... approval expiry = ... new worker says: my placement supports A3 my environment is eligible my tool contract matches my current epoch is valid Then the authority gateway can issue a short-lived scoped capability.
This preserves the Step 29 rule that capability and authority are separate.
Moving the computation does not automatically move permission.
6. Bind Every Checkpoint to a Behavioral Release A checkpoint created under one behavioral release may not be resumable under another.
Suppose the run was created with:
model-v7 prompt-bundle-18 router-policy-4 tool-contract-9 memory-schema-6 verifier-bundle-12 The replacement worker is running:
model-v8 prompt-bundle-19 tool-contract-10 memory-schema-7 That is not automatically a compatible continuation.
The handoff layer needs a compatibility decision:
@dataclass(frozen=True) class ResumeCompatibility: compatible: bool migration_required: bool reasons: tuple[str, ...] allowed_release_ids: tuple[str, ...] The safest default for long-running consequential work is often release pinning:
run starts on release R ↓ run remains on R until completion If migration to a new release is necessary, treat it as a behavioral migration—not a worker restart.
7. Preserve Observation Identity A replacement worker must know what the original worker actually observed.
That means storing evidence identity, not only summaries.
observation ├── source identity ├── retrieval/query parameters ├── timestamp ├── content hash ├── authority/freshness metadata └── derived interpretation references Why?
Because otherwise the new worker may silently re-query a live dependency and get different evidence.
That is not continuation.
It is re-observation.
Sometimes re-observation is exactly what we want.
But it must be explicit:
USE_RECORDED_OBSERVATION REVALIDATE_LIVE_STATE REFRESH_STALE_OBSERVATION Those are different operations with different semantics.
8. Separate Facts From Derived Agent State A useful portable state format distinguishes:
authoritative observations ↓ derived structured state ↓ agent hypotheses / plans Do not collapse all three into a prose scratchpad.
For example:
@dataclass(frozen=True) class PortableRunState: observations: tuple[str, ...] derived_facts_ref: str active_plan_ref: str | None candidate_refs: tuple[str, ...] unresolved_uncertainty_ref: str | None If the replacement worker uses a different model, it may reinterpret the same evidence.
That can be legitimate.
But the runtime should still preserve which parts were authoritative input and which parts were model-derived state.
9. Search State Is More Than the Winning Branch Advanced agents may maintain:
beam candidates, MCTS nodes, critic revisions, evolutionary populations, speculative branches, rejected alternatives. If migration keeps only the current winner, the algorithm has changed.
A portable search checkpoint may need:
node identities parent-child lineage visit counts scores verification status pruning reasons cancellation state budget consumed remaining frontier You do not always need all of this.
But if the scheduler claims the run can resume without changing search semantics, the required search state must be durable.
10. Budget State Must Move Too A migrated run should not receive a fresh budget accidentally.
That can break both reliability and cost controls.
Preserve:
model spend used model calls used search expansions used tool calls used browser time used sandbox time used verification reserve remaining wall-clock deadline retry budget Otherwise:
worker A spends 80% of budget ↓ migration ↓ worker B starts at 0% ↓ run quietly exceeds policy The budget ledger should belong to the logical run, not the worker.
11. Verification Evidence Must Follow Exact Artifact Identity Suppose candidate C1 was verified on worker A.
Worker B loads the checkpoint and normalizes whitespace, regenerates a file, or reserializes structured output.
Is it still C1?
Maybe.
Maybe not.
The verifier evidence should be bound to exact identity:
candidate_hash workspace_snapshot verifier_version verification_inputs verification_result If the candidate changes, the old verification should not automatically transfer.
The rule is:
Verified meaning must not be inferred from approximate artifact similarity.
Re-verify when identity changes.
12. Partial Side Effects Require Reconciliation The hardest handoff occurs after an ambiguous external action.
Example:
POST /deploy ↓ timeout ↓ worker dies Did deployment happen?
The replacement worker must not blindly retry.
Instead:
operation_id ↓ query authoritative external state ↓ COMMITTED / NOT_COMMITTED / UNKNOWN Then:
COMMITTED -> continue with postcondition verification NOT_COMMITTED -> retry if policy allows and idempotency holds UNKNOWN -> preserve UNKNOWN / escalate / reconcile further This is where idempotency keys and operation ledgers become essential.
Portable execution state must include side-effect lineage, not just reasoning state.
13. Browser Agents Need Special Treatment Browser state is notoriously difficult to migrate safely.
A browser worker may have:
cookies, CSRF tokens, DOM state, ephemeral sessions, partially completed forms, popup windows, anti-bot challenges, authentication state. Blindly serializing browser memory is rarely enough.
Useful resumability classes might be:
page can be re-opened from durable URL session can be reconstructed from scoped credentials workflow can restart from last confirmed server-side state workflow cannot be resumed safely For consequential browser workflows, prefer authoritative server-side state over assumptions about the old DOM.
14. Coding Agents Need Workspace Identity For coding work, checkpoint portability depends heavily on workspace identity.
A useful handoff may include:
repository URL base commit SHA worktree snapshot/hash uncommitted patch hash build environment identity dependency lockfiles test command definitions verification outputs If the repository has changed since the checkpoint:
old patch + new base is not automatically the same task state.
The runtime may need to:
resume against frozen snapshot rebase in sandbox re-run validation request new approval A migration that changes the base repository state should be recorded as a state transition, not hidden inside resume logic.
15. Provider Migration Is a Behavioral Change Moving from one machine to another running the same pinned model may be mostly operational.
Moving from a local model to a frontier API is different.
Even with the same prompt, the behavior changes.
So distinguish:
physical migration logical migration behavioral migration Physical migration Same behavioral release, compatible runtime, different worker.
Logical migration Same intended behavior, but state representation requires a compatibility transform.
Behavioral migration Model, prompt, router, tools, verifier semantics or other behavior-producing components change.
Behavioral migration should invoke Step 24 release rules and Step 30 competence rules.
Do not disguise it as failover.
16. Use a Handoff Protocol A safe handoff can be explicit:
1. quiesce source worker 2. finish or abort non-checkpointable local operations 3. seal checkpoint 4. persist checkpoint hash 5. record pending side effects 6. release/revoke source ownership 7. select compatible target placement 8. validate release/state/tool/verifier compatibility 9. acquire new lease + fencing epoch 10. revalidate external preconditions 11. restore logical state 12. resume from semantic boundary 13. verify first resumed transition Notice how much stronger this is than:
pickle.dump(agent) 17. Handoff Should Be Two-Phase A useful mental model is:
PREPARE HANDOFF ↓ checkpoint sealed compatibility evaluated target reserved COMMIT HANDOFF ↓ old authority fenced new owner epoch activated resume permitted If target preparation fails, the source worker may continue if still healthy.
If the handoff commits, the old worker must no longer be allowed to mutate.
This resembles transactional ownership transfer because that is effectively what it is.
18. Never Assume “Stopped” Means Stopped Step 19 already gave us the rule:
cancel requested ≠ underlying work actually stopped The same is true during migration.
A model request, browser action or tool execution may continue after the orchestration layer thinks the worker is quiesced.
Therefore the handoff record should track:
logical cancellation physical cancellation status possible outstanding external work operation IDs A new worker should not assume clean ownership merely because the old scheduler marked the task moved.
Fencing remains the ultimate protection for side effects.
19. State Schemas Need Compatibility Rules Portable state will evolve.
Eventually you will have:
checkpoint schema v3 checkpoint schema v4 checkpoint schema v5 Define directional compatibility.
v5 reader can read v4 v4 reader cannot read v5 Or require explicit migration:
migrate_v4_to_v5(checkpoint) The migration itself should be deterministic where possible and produce a new artifact with lineage back to the original.
Do not silently mutate old checkpoint records.
20. Some State Should Not Be Portable Portability is not always desirable.
Examples:
privileged credentials, hardware-bound secrets, region-restricted data, local security tokens, browser sessions prohibited from export, sensitive intermediate artifacts, provider-specific encrypted state. A checkpoint should be able to declare:
portable_to_regions = {eu-west} portable_to_providers = {local} requires_hardware_attestation = true contains_non_exportable_state = true Placement must respect that.
Step 34 chooses eligible targets.
Step 35 now gives it the state constraints required to decide whether migration is actually possible.
21. Handoff Can Reduce Authority Suppose the source worker was operating in a protected production enclave.
The only available replacement is a less trusted environment.
The correct result might be:
resume observation resume planning resume verification DO NOT resume mutation authority This is useful.
Handoff does not need to be all-or-nothing.
The target may inherit a reduced authority envelope.
That lets the task continue gathering evidence while waiting for a suitable mutation-capable placement.
22. Handoff Can Trigger Reverification Migration changes operational context.
Sometimes that alone invalidates previous assumptions.
For example:
a new region sees a different data replica, a new browser session changes server state, a new sandbox has different dependencies, a new tool version changes semantics, the external resource changed while the run was paused. So a resume policy might say:
if environment identity changed: revalidate state if artifact identity changed: reverify artifact if authority context changed: reauthorize if release changed: run compatibility gate Resume is not merely deserialization.
It is a guarded state transition.
23. Replay and Handoff Should Share the Same State Model Step 25 introduced replay manifests.
Portable execution should reuse the same provenance model rather than invent a parallel one.
Conceptually:
live run ↓ durable state + provenance ├── replay later └── resume elsewhere now That is powerful because the same artifacts support:
failover, debugging, audit, incident reconstruction, counterfactual replay, regression creation. Reliability infrastructure compounds when primitives are reused this way.
24. Observe Handoffs as First-Class Events A trajectory should record events such as:
checkpoint_started checkpoint_sealed handoff_requested source_quiesced source_fenced target_selected compatibility_passed compatibility_failed ownership_transferred state_restored revalidation_started revalidation_failed resume_started resume_verified handoff_aborted Useful fields include:
run_id checkpoint_id source_placement target_placement source_epoch target_epoch release_id state_schema reason compatibility_decision authority_before authority_after resume_boundary This makes migration visible in Step 26 incident forensics.
25. Measure Handoff Quality Useful metrics include:
handoff success rate resume failure rate checkpoint latency checkpoint size recovery time revalidation rate handoff-induced UNKNOWN rate duplicate-side-effect incidents stale-worker rejection count state incompatibility rate handoff rollback rate authority downgrade rate Also measure correctness:
verified success after handoff vs verified success without handoff A migration system that reduces infrastructure downtime but increases behavioral failure is not reliable.
26. Benchmark Against Simpler Alternatives Before building universal live migration, compare it against:
restart from beginning restart from last deterministic step retry on same placement wait for same resource pool checkpoint only at coarse boundaries human-assisted recovery Some workloads are cheap enough that restart is better than complex state portability.
This follows the series-wide rule:
Complexity must earn its cost empirically.
Do not build process migration because distributed-agent architecture diagrams look sophisticated.
Build it when long-running work, expensive evidence gathering, scarce resources or consequential side effects make restart materially worse.
27. Failure Injection Matters Test handoff under deliberately ugly conditions.
For example:
source dies before checkpoint seal source dies after seal but before lease release target dies after acquiring epoch old worker resumes after fencing checkpoint blob is missing checkpoint hash mismatches state schema is incompatible verifier bundle unavailable repository changed during pause browser session expired provider changed model alias external side effect timed out approval expired during migration budget expires during handoff region policy changes mid-run For every case, ask:
Can duplicate mutation occur? Can stale evidence be accepted? Can authority expand accidentally? Can verification attach to the wrong artifact? Can budget reset? Can UNKNOWN be hidden? Those are more important than whether the worker successfully deserializes JSON.
28. A Minimal Handoff Decision The first implementation does not need a learned migration controller.
A deterministic compatibility gate is a better starting point.
from dataclasses import dataclass @dataclass(frozen=True) class HandoffDecision: allowed: bool resume_mode: str authority_ceiling: str requires_revalidation: bool requires_reverification: bool reasons: tuple[str, ...] def evaluate_handoff(checkpoint, target): reasons = [] if checkpoint.release_id not in target.supported_release_ids: reasons.append("release_incompatible") if checkpoint.state_schema_version not in target.readable_state_schemas: reasons.append("state_schema_incompatible") if not target.can_access(checkpoint.required_data_scope): reasons.append("data_scope_incompatible") if not target.has_required_verifiers(checkpoint.required_verifiers): reasons.append("verifier_unavailable") if reasons: return HandoffDecision( allowed=False, resume_mode="NONE", authority_ceiling="NONE", requires_revalidation=False, requires_reverification=False, reasons=tuple(reasons), ) return HandoffDecision( allowed=True, resume_mode="CHECKPOINT", authority_ceiling=target.authority_ceiling, requires_revalidation=True, requires_reverification=False, reasons=(), ) Later, placement or migration policies can become more adaptive.
But the hard compatibility boundaries should remain deterministic.
29. The Full Architecture The resulting runtime now looks like this:
logical agent run ↓ durable execution state ↓ semantic checkpoint ↓ handoff coordinator ↓ compatibility gate ↓ capability-aware placement ↓ ownership transfer ↓ lease + fencing epoch ↓ state restore ↓ external-state revalidation ↓ resume ↓ verification Around it sit the systems built earlier in the series:
behavioral release manifest competence envelope authority policy SLO / error budget replay provenance incident forensics capability dependency graph placement policy The checkpoint is therefore not an isolated runtime feature.
It is another point where the entire reliability architecture comes together.
30. The Real Goal Is Continuity of Meaning There is a tempting implementation mindset here:
Can we serialize enough state to restart the program elsewhere?
That is the wrong goal.
The stronger question is:
Can the replacement execution continue the same logical task under explicit, compatible evidence, ownership, authority and verification semantics?
Those are not equivalent.
A migrated process can be technically alive while the original task has already lost its meaning.
That is why portable execution state must preserve more than memory.
It must preserve the contract of continuation.
The rule is worth repeating:
A task may move. Its meaning, evidence, and authority must not move implicitly.
What Comes Next? Once execution state is portable, another problem appears.
A long-running agent may span hours or days.
During that time:
repository state changes, web evidence becomes stale, dependencies are updated, approvals expire, model aliases move, policies change, other agents modify shared resources, users change their intent. A checkpoint can preserve what the agent knew.
It cannot guarantee that what it knew is still true.
The next stage is therefore temporal consistency and stale-state management:
How does an agent know which assumptions must be refreshed before it continues acting?
That means leases not only on workers, but on facts.
Freshness budgets.
Read-set tracking.
Optimistic concurrency.
Precondition revalidation.
Conflict detection.
And explicit invalidation when the world moves underneath a long-running plan.
Because in a real agent system, the hardest state problem is not merely moving state.
It is knowing when that state has expired.