Can Your Agent Coordinate Across Machines Without Duplicating Work? Use Leases, Idempotency and Fencing
Can Your Agent Coordinate Across Machines Without Duplicating Work?
A single-process agent can already be complicated.
It can plan.
It can search.
It can launch speculative branches.
It can cancel losing work.
It can verify outcomes.
Then you move that work onto multiple workers.
Now a new class of failure appears.
Two workers both believe they own the same task.
One worker pauses for thirty seconds.
Another worker assumes it died and takes over.
The first worker wakes up and continues writing.
A retry sends the same email twice.
A duplicated browser worker submits the same form twice.
Two coding workers both push incompatible changes.
A worker renews an expired lease too late.
A message is delivered again after the original acknowledgement was lost.
The model may have made the correct decision every time.
The distributed runtime can still corrupt the result.
That leads to the central rule of this post:
Once agent work leaves a single process, correctness depends on ownership, idempotency, fencing and recovery—not just reasoning quality.
The important shift is this:
single process
↓
who should do this work?
becomes
distributed runtime
↓
who currently owns this work?
↓
is that ownership still valid?
↓
can a stale worker still mutate state?
↓
what happens if the same operation runs twice?
This post is about making those questions explicit.
The distributed-agent problem
Suppose a coordinator assigns a coding task to worker A.
coordinator
↓
worker A
↓
edit repository
↓
run tests
↓
commit result
Worker A stops sending heartbeats.
The coordinator waits for a timeout and assigns the task to worker B.
coordinator
├── worker A ?
└── worker B active
Worker B starts from the latest repository state.
Then worker A comes back.
Now both workers believe they are allowed to finish.
Without stronger coordination, you have split ownership.
That can produce:
- duplicate mutations,
- stale writes,
- double billing,
- duplicate emails,
- repeated deployments,
- conflicting commits,
- duplicated browser submissions,
- corrupted checkpoints,
- and misleading completion events.
The first requirement is therefore not intelligence.
It is exclusive, time-bounded ownership.
A lease is temporary ownership
A lease says:
Worker A owns task T until time X, unless the lease is renewed.
For example:
task_id: repair-482
owner: worker-a
lease_epoch: 17
expires_at: 11:14:30
Worker A must periodically renew the lease.
acquire
↓
work
↓
heartbeat / renew
↓
work
↓
heartbeat / renew
↓
complete
If renewal stops, another worker may eventually acquire ownership.
This is useful because distributed runtimes cannot perfectly distinguish:
worker crashed
worker paused
network partition
scheduler delay
GC pause
machine overloaded
heartbeat lost
A lease avoids waiting forever.
But a lease alone does not solve stale-worker writes.
That requires fencing.
Why lease expiry is not enough
Imagine this sequence:
1. worker A gets lease epoch 17
2. worker A pauses
3. lease expires
4. worker B gets lease epoch 18
5. worker B begins work
6. worker A resumes
7. worker A writes stale result
Worker A may not know its lease expired.
It may still have credentials.
It may still have an open database connection.
It may still be able to call the deployment API.
The runtime therefore needs a way for the resource itself to reject stale owners.
That mechanism is a fencing token.
Fencing tokens prevent stale owners from committing
Every successful lease acquisition gets a monotonically increasing token.
epoch 17 -> worker A
epoch 18 -> worker B
Every mutation includes that token.
write(task_id="repair-482", epoch=18, payload=...)
The receiving system remembers the highest accepted epoch.
If worker A later tries:
write(task_id="repair-482", epoch=17, payload=...)
it is rejected.
The important point is:
Ownership validity is enforced at the mutation boundary, not merely trusted inside the worker.
That is much stronger than saying:
if lease.is_valid():
do_write()
because the lease can expire between the check and the write.
This is the distributed equivalent of a time-of-check/time-of-use bug.
Minimal lease model
A simple lease record might look like this:
from dataclasses import dataclass
from datetime import datetime
@dataclass
class Lease:
task_id: str
owner_id: str
epoch: int
expires_at: datetime
Acquisition should be atomic.
Conceptually:
read current lease
↓
if absent or expired
↓
increment epoch
↓
write owner + expiry + epoch atomically
The implementation may use:
- a transactional database row,
- compare-and-swap,
- a strongly consistent key-value store,
- a queue visibility timeout plus external fencing,
- or a dedicated coordination service.
The mechanism matters more than the product.
Heartbeats are evidence of liveness, not proof of safety
Heartbeats help answer:
Is this worker probably still active?
They do not answer:
Is this worker still authorized to mutate shared state?
Those are different questions.
heartbeat -> liveness signal
lease -> ownership window
fence -> mutation authority
Do not collapse them into one mechanism.
A worker can send heartbeats while holding stale state.
A heartbeat can be delayed.
A lease renewal can race with reassignment.
The mutation boundary still needs an ownership check.
Why “exactly once” is usually the wrong goal
Distributed systems often retry.
Messages are redelivered.
Workers crash after doing work but before acknowledging completion.
Networks duplicate or delay messages.
Suppose an agent sends an invoice email.
1. worker sends email
2. process crashes
3. completion acknowledgement never reaches coordinator
4. task is retried
5. second worker sends email again
The system may have executed the operation twice even though the coordinator saw only one completion.
The safer mental model is:
at-least-once delivery
+
idempotent operation
+
state verification
instead of pretending the whole pipeline executes exactly once.
Idempotency means retries do not create new effects
An operation is idempotent when repeated execution with the same logical request produces the same external effect.
For example:
create_payment(request_id=abc123)
should not create two payments if retried.
The external service stores the idempotency key:
abc123 -> payment_987
A repeated request returns the existing result.
The same pattern applies to agent systems.
Useful idempotency keys may include:
task_id
operation_id
side_effect_type
target_resource
input_state_version
The key must identify the logical operation, not the worker attempt.
This is wrong:
idempotency_key = worker_attempt_id
because every retry gets a new key.
This is better:
idempotency_key = task_id + operation_name + target_id
provided those values uniquely define the intended effect.
Attempt identity and operation identity are different
This distinction is essential.
logical operation
↓
operation_id = deploy-service-v7
attempt 1
attempt 2
attempt 3
Retries should share the same operation identity.
But each physical execution attempt should have its own attempt identity.
operation_id: deploy-service-v7
attempt_id: 1
worker: A
operation_id: deploy-service-v7
attempt_id: 2
worker: B
That lets observability answer both:
- which logical effect was intended?
- how many physical attempts occurred?
Idempotency does not make everything safe
Some operations are naturally idempotent.
set desired replicas = 5
Running it twice generally produces the same desired state.
Others are not.
increment balance by 10
send email
append comment
submit form
create ticket
trigger deployment
For non-idempotent operations, the runtime may need:
- idempotency keys,
- compare-and-swap,
- transaction records,
- deduplication tables,
- state-machine transitions,
- or an outbox/inbox pattern.
The design question is not:
Can the worker retry?
It is:
Can the external effect be safely retried?
Use state transitions instead of vague completion flags
A boolean like this is often too weak:
completed = true
A distributed task usually has several meaningful states.
PENDING
LEASED
RUNNING
VERIFYING
COMMITTING
SUCCEEDED
FAILED
UNKNOWN
You may also need:
CANCEL_REQUESTED
CANCELLED
LEASE_EXPIRED
RETRYABLE_FAILURE
PERMANENT_FAILURE
The transitions should be explicit.
PENDING
↓ acquire
LEASED
↓ start
RUNNING
↓ work complete
VERIFYING
↓ verifier PASS
COMMITTING
↓ fenced/idempotent commit
SUCCEEDED
This is much easier to reason about than arbitrary worker-written flags.
Completion itself must be fenced
Suppose worker A finishes after losing ownership.
It should not be allowed to mark the task SUCCEEDED.
Completion updates need the same ownership discipline as mutations.
Conceptually:
def complete_task(task_id, owner_id, epoch, result):
task = load_task(task_id)
if epoch < task.current_epoch:
raise StaleOwner()
if owner_id != task.current_owner:
raise NotOwner()
task.status = "SUCCEEDED"
task.result = result
save(task)
The exact implementation will vary.
The principle does not.
Retries require a classification policy
Not every failure should be retried.
A useful taxonomy is:
transient infrastructure failure
rate limit
worker crash
network timeout
stale lease
invalid input
verification failure
policy violation
permanent external error
Then the retry policy can be explicit.
network timeout -> retry
rate limit -> backoff + retry
worker crash -> reacquire
stale lease -> stop stale worker
invalid input -> do not retry blindly
verification failure -> revise/replan
policy violation -> fail closed
“Retry” is not a universal recovery mechanism.
Retries need backoff and jitter
If 500 distributed workers all retry at the same moment, the recovery mechanism becomes another outage.
Use bounded backoff.
attempt 1 -> wait ~1s
attempt 2 -> wait ~2s
attempt 3 -> wait ~4s
attempt 4 -> wait ~8s
Add jitter so workers do not synchronize.
Also cap the retry budget.
max attempts
max elapsed time
max cost
max external calls
A retry policy is another budget policy.
Distributed cancellation is also not instantaneous
Step 19 distinguished logical cancellation from physical cancellation.
That becomes even more important across machines.
coordinator marks CANCEL_REQUESTED
↓
message travels
↓
worker eventually observes cancellation
↓
worker stops
↓
resources released
During that interval the worker may continue consuming:
- tokens,
- CPU,
- GPU,
- tool calls,
- browser sessions,
- database connections,
- external API quota.
The worker should therefore poll or receive cancellation signals at safe checkpoints.
But cancellation still does not authorize a stale worker to commit.
The fencing boundary remains necessary.
Ownership and cancellation should be visible in the trajectory
The Step 13 observability model should now include distributed events.
lease_acquired
lease_renewed
lease_expired
lease_reassigned
heartbeat_sent
heartbeat_missed
worker_started
worker_retried
worker_cancel_requested
worker_cancel_acknowledged
fence_rejected
idempotency_hit
commit_started
commit_completed
Each event should include:
task_id
operation_id
attempt_id
worker_id
lease_epoch
state_id
policy_version
resource_target
cost
outcome
That lets you distinguish:
model failure
vs
worker failure
vs
ownership race
vs
retry duplication
vs
stale commit
vs
external service failure
A distributed-agent runtime should think in ownership domains
Not every action needs a global lock.
That would destroy concurrency.
Instead define the smallest useful ownership domain.
Examples:
repository branch
customer ticket
browser session
dataset partition
deployment target
research claim
search-tree node
If two workers can safely operate on independent domains, they should.
The rule is:
Coordinate only where effects can conflict.
This preserves throughput while protecting consistency.
Coding agents
A distributed coding system may run several repair workers.
A good isolation structure might be:
task
├── worktree A
├── worktree B
└── worktree C
Each worker can freely:
- edit,
- run tests,
- inspect code,
- build artifacts.
But the shared mutation boundary is the repository branch or PR.
Commit/application should require:
current task lease
+
fencing epoch
+
base commit still valid
+
verification PASS
Before applying a result:
re-read current branch head
compare with speculative base
revalidate patch
run required tests
commit with current ownership token
A stale worker should never be able to overwrite a newer accepted repair.
Research agents
Distributed research workers are often safer because much of the work is read-only.
Workers can independently retrieve:
- papers,
- webpages,
- datasets,
- quotes,
- counterexamples,
- historical context.
But shared evidence stores still need deduplication and provenance.
Two workers may discover the same source.
That should not create two independent evidence votes.
Use source identity and claim identity.
source_id
claim_id
retrieval_attempt_id
worker_id
Idempotent evidence insertion is better than counting duplicate retrievals as stronger evidence.
Browser agents
Browser automation can be dangerous under retries.
Read-only work is usually easy to parallelize.
open page
inspect DOM
collect options
compare prices
Mutation is harder.
submit form
place order
send message
book reservation
Those actions should have strong operation identities.
For example:
operation_id = reservation(user, restaurant, date, party_size)
Before retrying, inspect whether the external state already reflects the requested action.
A timeout does not necessarily mean the action failed.
This is one of the most common distributed-agent mistakes:
request timed out
↓
assume failure
↓
retry mutation
↓
duplicate external effect
A timeout means:
Outcome unknown.
That should trigger state observation before blind retry.
Data agents
Distributed data agents often process partitions.
The natural ownership domain may be:
(dataset_id, partition_id, version)
Each worker should write outputs under deterministic identifiers.
output_key = hash(dataset_version, partition_id, transform_version)
Retries can then detect existing outputs.
But verify that the existing artifact matches the intended transform version.
Idempotency without version binding can silently reuse stale results.
DevOps agents
Distributed DevOps agents need the strongest boundaries.
Suppose two workers both decide to restart a service.
Or two remediation workers both scale a cluster.
The outcome may be worse than the original incident.
Use explicit operation identities and fencing at the control-plane boundary.
incident_id
service_id
remediation_action
resource_version
lease_epoch
Before executing:
verify current incident state
verify current resource version
verify ownership
execute idempotent desired-state mutation
verify postcondition
Desired-state APIs are especially useful.
set replicas = 8
is usually safer than:
add 3 replicas
because retries are easier to reason about.
Queue delivery and task ownership are not the same thing
A queue may make a message temporarily invisible after a worker receives it.
That is useful.
It is not necessarily sufficient for distributed correctness.
A worker can retain credentials after visibility expires.
A message can be redelivered.
External mutations can outlive queue ownership.
So treat queue visibility as one ownership signal, not universal fencing.
queue visibility
+
lease/epoch
+
idempotent mutation boundary
+
verification
is much stronger.
The outbox pattern prevents lost side effects
A classic failure looks like this:
1. update database: task succeeded
2. crash
3. send completion event never happens
Or the reverse:
1. send completion event
2. crash
3. database update never happens
Now external observers disagree about task state.
One solution is an outbox.
Inside the same transaction that updates task state, write an event record.
transaction:
update task status
insert outbox event
commit
A separate publisher sends outbox events and marks them delivered.
Delivery itself may be at least once, so downstream consumers must deduplicate.
This is boring distributed-systems engineering.
That is exactly why advanced agent runtimes need it.
The inbox pattern deduplicates incoming work
If consumers may receive the same event more than once, record processed message IDs.
Conceptually:
receive message M
↓
was M already processed?
├─ yes -> return stored result
└─ no -> process + record atomically
Again:
At-least-once delivery plus idempotent consumption is often more realistic than pretending duplicates never occur.
Crash recovery needs durable checkpoints
Long-running agent tasks may survive multiple workers.
Do not rely on one worker’s in-memory conversation as the source of truth.
Persist meaningful checkpoints.
For example:
task state
accepted plan version
completed steps
current environment version
verified evidence
memory references
budget consumed
pending operations
A replacement worker should be able to reconstruct the task from durable state.
This connects directly to the earlier distinction:
runtime state ≠ chat history
Distributed execution makes that distinction mandatory.
Checkpoints must be versioned
A checkpoint should identify the exact environment it describes.
repo_commit
browser_session_version
dataset_version
deployment_revision
policy_version
model_version
Otherwise a new worker may resume from logically stale state.
Before continuing:
load checkpoint
↓
observe current external state
↓
compare versions
↓
resume / replan / invalidate
Do not let workers invent ownership from memory
This is worth making explicit.
A model may say:
I am still responsible for this task.
That is irrelevant.
Ownership is a runtime fact.
model memory
≠
lease state
Workers should obtain ownership from an authoritative coordination store.
This follows the source-of-truth rule from the uncertainty and EVI posts.
Exact state should be observed directly when possible.
Distributed agents need reentrancy
A task handler should tolerate being entered more than once.
That means:
- reconstruct state from durable records,
- detect already-completed operations,
- verify external state,
- reuse idempotency keys,
- avoid assuming local memory is unique,
- and make retries explicit.
A useful question is:
If this worker crashes after any line, what happens when another worker starts again?
If the answer is “we hope it doesn’t happen,” the design is not ready for distributed execution.
Side effects need a commit protocol
For consequential tasks, separate preparation from commit.
prepare
↓
verify ownership
↓
verify current state
↓
verify candidate
↓
reserve idempotency key
↓
commit side effect
↓
verify postcondition
This resembles Step 19’s speculative/commit boundary.
The difference is that now the boundary must survive:
- worker crashes,
- network partitions,
- reassignment,
- duplicate attempts,
- delayed messages.
A compact distributed task schema
A practical task record might include:
from dataclasses import dataclass
from typing import Optional
@dataclass
class DistributedTask:
task_id: str
status: str
owner_id: Optional[str]
lease_epoch: int
lease_expires_at: Optional[float]
operation_id: str
attempt_count: int
state_version: str
policy_version: str
result_ref: Optional[str] = None
verification_status: str = "UNKNOWN"
This is not enough for every production system.
It is enough to expose the important concepts.
Worker loop from first principles
Conceptually:
async def worker_loop(worker_id, store, executor):
while True:
lease = await store.acquire_next(worker_id)
if lease is None:
await sleep_briefly()
continue
try:
task = await store.load(lease.task_id)
state = await executor.observe_current_state(task)
if await executor.already_completed(task, state):
await store.complete_if_owner(
task_id=task.task_id,
owner_id=worker_id,
epoch=lease.epoch,
result=state.existing_result,
)
continue
result = await executor.run(task, lease)
verification = await executor.verify(task, result)
if verification.status != "PASS":
await store.fail_or_retry_if_owner(
task.task_id,
worker_id,
lease.epoch,
verification,
)
continue
await executor.commit_with_fence(
task=task,
result=result,
epoch=lease.epoch,
operation_id=task.operation_id,
)
await store.complete_if_owner(
task.task_id,
worker_id,
lease.epoch,
result,
)
except StaleOwner:
# Another worker now owns the task.
continue
The key idea is not the syntax.
It is that ownership is checked repeatedly at boundaries that matter.
Measure distributed coordination separately from agent quality
If a run fails, do not automatically blame the model.
Useful metrics include:
lease acquisition latency
lease-expiry rate
heartbeat miss rate
reassignment rate
duplicate-attempt rate
stale-owner rejection rate
idempotency-hit rate
duplicate-side-effect rate
retry success rate
retry amplification
post-cancel compute
checkpoint-resume success
commit conflict rate
UNKNOWN-after-timeout rate
Then separately measure:
verified task success
false success
cost per verified success
critical-path latency
This lets you answer:
Is the agent bad, or is the distributed runtime bad?
Duplicate work is not always wrong
Step 19 deliberately launched duplicate speculative work when it reduced latency or increased candidate diversity.
Distributed duplication can therefore be intentional.
The key distinction is:
intentional duplicate computation
≠
duplicate side effect
Two workers may both investigate a bug.
Only one should merge the winning fix.
Two workers may retrieve the same source.
The evidence store should deduplicate the source.
Two workers may simulate a deployment plan.
Only the selected plan should mutate production.
The distributed runtime must know which duplication is allowed and where exclusivity begins.
Ownership can be hierarchical
Large agent systems may have nested work.
run
├── planning task
├── research task
│ ├── source A
│ └── source B
└── implementation task
├── worktree A
└── worktree B
You may not need one global lease.
Each node can have independent ownership.
But parent cancellation and budget policies must propagate.
Useful relationships include:
parent_task_id
root_run_id
operation_id
lease_epoch
attempt_id
This makes distributed traces reconstructable as graphs.
Fencing must reach the real side-effect boundary
A common mistake is to validate fencing only in the coordinator database.
Example:
coordinator accepts worker B as owner
worker A still has cloud credentials
worker A calls cloud API directly
The stale worker bypassed the fence.
When possible, enforce fencing where the side effect happens.
Examples:
- compare resource version before mutation,
- pass generation/epoch to transactional store,
- require current deployment revision,
- use conditional writes,
- route privileged mutations through a fenced gateway.
If the underlying API cannot enforce fencing, the risk should be explicit.
Time is not a perfect coordination primitive
Lease expiry uses time.
Do not assume clocks are perfectly synchronized.
Prefer the coordination store’s authoritative lease state over each worker’s local interpretation of wall-clock time.
Also include renewal margins.
A worker should not begin a long irreversible commit when the lease is about to expire.
For example:
remaining lease: 500 ms
estimated commit: 8 s
The correct action is probably:
renew first
or stop.
Lease duration is a policy parameter
Too short:
- healthy workers lose ownership,
- retries increase,
- duplicate computation rises.
Too long:
- recovery from dead workers is slow,
- tasks remain unavailable longer.
Measure the distribution of task step durations and heartbeat delays.
Then choose a lease policy based on evidence.
You may use different lease durations by task class.
read-only retrieval: short
long test suite: longer
deployment commit: explicit protected window
Again:
One global timeout is rarely a good architecture.
Retry amplification can destroy a system
Suppose a dependency slows down.
Workers time out.
Each task retries.
Concurrency doubles.
The dependency becomes slower.
More tasks retry.
Now the recovery loop amplifies the outage.
slow dependency
↓
timeouts
↓
retries
↓
more load
↓
slower dependency
Protect the runtime with:
- concurrency limits,
- retry budgets,
- backoff,
- jitter,
- circuit breakers,
- queue backpressure,
- and cancellation.
This connects directly to Step 16’s dynamic budget scheduler.
Retries consume the same finite resource envelope as reasoning and verification.
Preserve verification under retries
A dangerous failure mode is:
attempt 1 runs full verification
attempt 2 is a retry path
attempt 2 skips verification to save time
attempt 2 commits
Retry paths must not silently weaken acceptance criteria.
The protected verification reserve from Step 16 still applies.
If anything, distributed ambiguity makes verification more important.
UNKNOWN is essential after ambiguous side effects
Suppose a deployment request times out.
The runtime does not know whether production accepted it.
Do not classify that as FAIL immediately.
Use:
UNKNOWN
Then observe current state.
timeout
↓
UNKNOWN
↓
inspect deployment state
↓
PASS / FAIL / still UNKNOWN
This is a direct application of the verification and uncertainty posts.
Blind retry can be worse than waiting for evidence.
Distributed correctness can simplify agent behavior
Once ownership, retries, state, and commit semantics are explicit, the model has fewer things to reason about.
That is good.
The model should not decide:
- whether it still owns the task,
- whether a duplicate operation already happened,
- whether retry budgets remain,
- whether a stale worker may commit,
- whether a message was already processed.
Those are runtime responsibilities.
The model can focus on the domain problem.
This follows the same principle used throughout the series:
Move deterministic control out of the language model whenever the runtime can represent it directly.
A distributed agent architecture
Putting the pieces together:
task queue
|
coordinator
|
lease / epoch store
/ | \
worker A worker B worker C
| | |
isolated work / observations
\ | /
\ | /
candidate/evidence store
|
selector/verifier
|
fenced commit gateway
|
external system
|
postcondition check
The control plane owns:
- task assignment,
- leases,
- epochs,
- retries,
- cancellation,
- budgets,
- policy versions.
Workers own temporary execution.
The commit gateway owns consequential mutation.
The verifier owns acceptance evidence.
That separation is powerful.
Failure taxonomy
Useful distributed-agent failure labels include:
LEASE_EXPIRED
STALE_OWNER
DUPLICATE_ATTEMPT
DUPLICATE_SIDE_EFFECT
IDEMPOTENCY_MISS
FENCE_BYPASS
HEARTBEAT_FAILURE
RETRY_EXHAUSTED
RETRY_AMPLIFICATION
CHECKPOINT_STALE
CHECKPOINT_CORRUPT
COMMIT_CONFLICT
CANCEL_NOT_OBSERVED
POST_CANCEL_WORK
AMBIGUOUS_SIDE_EFFECT
OWNERSHIP_SPLIT
Do not collapse all of these into:
agent error
Each implies a different remediation.
Failure injection
Distributed correctness should be tested deliberately.
Inject failures such as:
Worker crashes after external mutation but before completion acknowledgement
Expected:
- retry detects existing effect,
- no duplicate effect,
- task eventually reconciles.
Worker pauses until lease expires, then resumes
Expected:
- new owner gets higher epoch,
- stale worker’s commit is rejected.
Completion event delivered twice
Expected:
- consumer deduplicates.
Network timeout after mutation request
Expected:
- outcome becomes
UNKNOWN, - runtime observes authoritative state before retry.
Coordinator sends cancellation during a long model call
Expected:
- logical cancellation recorded,
- physical stop attempted,
- post-cancel cost measured,
- stale result cannot commit.
Checkpoint references old environment version
Expected:
- resume detects mismatch,
- task replans or invalidates stale state.
These tests are often more valuable than another hundred happy-path agent benchmarks.
Benchmark the distributed runtime fairly
Compare:
single-worker baseline
fixed worker pool
lease-based worker pool
distributed speculative pool
adaptive worker pool
Under the same task set.
Measure:
verified success
false success
duplicate side effects
cost
p50/p95 latency
recovery time
retry count
stale-owner rejection
idempotency hit rate
The distributed system should earn its complexity.
If one worker already handles the throughput and latency requirements, do not build a miniature distributed-systems research project for aesthetics.
When do you actually need distributed agent execution?
You probably do not need it when:
- tasks are short,
- concurrency is low,
- one machine has enough capacity,
- failures can simply restart the whole run,
- external side effects are rare,
- queue delay is acceptable.
You may need it when:
- many independent tasks must run concurrently,
- model/tool workloads exceed one host,
- tasks are long-running,
- workers fail independently,
- specialized hardware is distributed,
- jobs must survive process restarts,
- speculative branches need many executors,
- throughput or latency requires horizontal scaling.
The rule remains:
Add distributed coordination because measured workload requires it, not because distributed agents sound more advanced.
Ten rules for distributed agents
- Treat ownership as explicit runtime state.
- Use leases for temporary ownership, not eternal locks.
- Use fencing tokens to reject stale owners at mutation boundaries.
- Design for retries and duplicate delivery.
- Make side effects idempotent whenever possible.
- Separate logical operation identity from physical attempt identity.
- Persist checkpoints; do not trust one worker’s memory.
- Use
UNKNOWNfor ambiguous external outcomes. - Keep verification and safety requirements intact across retries.
- Measure distributed coordination failures separately from model failures.
The deeper architectural lesson
Advanced agents eventually stop looking like a prompt around a model.
They start looking like serious software systems.
They need:
- scheduling,
- state machines,
- ownership,
- retries,
- observability,
- transactions,
- isolation,
- verification,
- recovery,
- versioning.
That is not accidental.
The more autonomy the system receives, the more important ordinary systems engineering becomes.
The final control loop now looks like this:
request
↓
policy + budget
↓
distributed task graph
↓
leases + ownership
↓
workers / speculative branches
↓
evidence + candidates
↓
selection
↓
verification
↓
fenced idempotent commit
↓
postcondition verification
↓
trajectory + durable checkpoint
The model is still important.
But the model is no longer the architecture.
What comes next?
Distributed execution creates another problem.
Now your system may have:
- hundreds of workers,
- many concurrent task graphs,
- shared model capacity,
- rate-limited tools,
- GPU pools,
- browser pools,
- database connections,
- and multiple tenants competing for the same resources.
Per-task budgeting is no longer enough.
The next question becomes:
How should an agent platform schedule many competing runs fairly under shared capacity constraints?
That means moving from per-agent budgets to global admission control, quotas, priorities, fairness and backpressure.
That is the next stage.
