# Advanced Agents From First Principles 20: Can Your Agent Coordinate Across Machines Without Duplicating Work? Use Leases, Idempotency and Fencing
# Book: Advanced Agents From First Principles
# Chapter URL: https://programmer.ie/books/advanced-agents-from-first-principles/20-chapter/
# Prose fingerprint (sha256, normalized): 8a401654c48a6f4c1913416bafd89f7aba38f0a7e73cf95ade47d4e892942a55
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

Can Your Agent Coordinate Across Machines Without Duplicating Work? A single-process agent can already be complicated.
It can plan.
It can search.
It can launch speculative branches.
It can cancel losing work.
It can verify outcomes.
Then you move that work onto multiple workers.
Now a new class of failure appears.
Two workers both believe they own the same task.
One worker pauses for thirty seconds.
Another worker assumes it died and takes over.
The first worker wakes up and continues writing.
A retry sends the same email twice.
A duplicated browser worker submits the same form twice.
Two coding workers both push incompatible changes.
A worker renews an expired lease too late.
A message is delivered again after the original acknowledgement was lost.
The model may have made the correct decision every time.
The distributed runtime can still corrupt the result.
That leads to the central rule of this post:
Once agent work leaves a single process, correctness depends on ownership, idempotency, fencing and recovery—not just reasoning quality.
The important shift is this:
single process ↓ who should do this work? becomes distributed runtime ↓ who currently owns this work? ↓ is that ownership still valid? ↓ can a stale worker still mutate state? ↓ what happens if the same operation runs twice? This post is about making those questions explicit.
The distributed-agent problem Suppose a coordinator assigns a coding task to worker A.
coordinator ↓ worker A ↓ edit repository ↓ run tests ↓ commit result Worker A stops sending heartbeats.
The coordinator waits for a timeout and assigns the task to worker B.
coordinator ├── worker A ? └── worker B active Worker B starts from the latest repository state.
Then worker A comes back.
Now both workers believe they are allowed to finish.
Without stronger coordination, you have split ownership.
That can produce:
duplicate mutations, stale writes, double billing, duplicate emails, repeated deployments, conflicting commits, duplicated browser submissions, corrupted checkpoints, and misleading completion events. The first requirement is therefore not intelligence.
It is exclusive, time-bounded ownership.
A lease is temporary ownership A lease says:
Worker A owns task T until time X, unless the lease is renewed.
For example:
task_id: repair-482 owner: worker-a lease_epoch: 17 expires_at: 11:14:30 Worker A must periodically renew the lease.
acquire ↓ work ↓ heartbeat / renew ↓ work ↓ heartbeat / renew ↓ complete If renewal stops, another worker may eventually acquire ownership.
This is useful because distributed runtimes cannot perfectly distinguish:
worker crashed worker paused network partition scheduler delay GC pause machine overloaded heartbeat lost A lease avoids waiting forever.
But a lease alone does not solve stale-worker writes.
That requires fencing.
Why lease expiry is not enough Imagine this sequence:
1. worker A gets lease epoch 17 2. worker A pauses 3. lease expires 4. worker B gets lease epoch 18 5. worker B begins work 6. worker A resumes 7. worker A writes stale result Worker A may not know its lease expired.
It may still have credentials.
It may still have an open database connection.
It may still be able to call the deployment API.
The runtime therefore needs a way for the resource itself to reject stale owners.
That mechanism is a fencing token.
Fencing tokens prevent stale owners from committing Every successful lease acquisition gets a monotonically increasing token.
epoch 17 -> worker A epoch 18 -> worker B Every mutation includes that token.
write(task_id="repair-482", epoch=18, payload=...) The receiving system remembers the highest accepted epoch.
If worker A later tries:
write(task_id="repair-482", epoch=17, payload=...) it is rejected.
The important point is:
Ownership validity is enforced at the mutation boundary, not merely trusted inside the worker.
That is much stronger than saying:
if lease.is_valid(): do_write() because the lease can expire between the check and the write.
This is the distributed equivalent of a time-of-check/time-of-use bug.
Minimal lease model A simple lease record might look like this:
from dataclasses import dataclass from datetime import datetime @dataclass class Lease: task_id: str owner_id: str epoch: int expires_at: datetime Acquisition should be atomic.
Conceptually:
read current lease ↓ if absent or expired ↓ increment epoch ↓ write owner + expiry + epoch atomically The implementation may use:
a transactional database row, compare-and-swap, a strongly consistent key-value store, a queue visibility timeout plus external fencing, or a dedicated coordination service. The mechanism matters more than the product.
Heartbeats are evidence of liveness, not proof of safety Heartbeats help answer:
Is this worker probably still active?
They do not answer:
Is this worker still authorized to mutate shared state?
Those are different questions.
heartbeat -> liveness signal lease -> ownership window fence -> mutation authority Do not collapse them into one mechanism.
A worker can send heartbeats while holding stale state.
A heartbeat can be delayed.
A lease renewal can race with reassignment.
The mutation boundary still needs an ownership check.
Why “exactly once” is usually the wrong goal Distributed systems often retry.
Messages are redelivered.
Workers crash after doing work but before acknowledging completion.
Networks duplicate or delay messages.
Suppose an agent sends an invoice email.
1. worker sends email 2. process crashes 3. completion acknowledgement never reaches coordinator 4. task is retried 5. second worker sends email again The system may have executed the operation twice even though the coordinator saw only one completion.
The safer mental model is:
at-least-once delivery + idempotent operation + state verification instead of pretending the whole pipeline executes exactly once.
Idempotency means retries do not create new effects An operation is idempotent when repeated execution with the same logical request produces the same external effect.
For example:
create_payment(request_id=abc123) should not create two payments if retried.
The external service stores the idempotency key:
abc123 -> payment_987 A repeated request returns the existing result.
The same pattern applies to agent systems.
Useful idempotency keys may include:
task_id operation_id side_effect_type target_resource input_state_version The key must identify the logical operation, not the worker attempt.
This is wrong:
idempotency_key = worker_attempt_id because every retry gets a new key.
This is better:
idempotency_key = task_id + operation_name + target_id provided those values uniquely define the intended effect.
Attempt identity and operation identity are different This distinction is essential.
logical operation ↓ operation_id = deploy-service-v7 attempt 1 attempt 2 attempt 3 Retries should share the same operation identity.
But each physical execution attempt should have its own attempt identity.
operation_id: deploy-service-v7 attempt_id: 1 worker: A operation_id: deploy-service-v7 attempt_id: 2 worker: B That lets observability answer both:
which logical effect was intended? how many physical attempts occurred? Idempotency does not make everything safe Some operations are naturally idempotent.
set desired replicas = 5 Running it twice generally produces the same desired state.
Others are not.
increment balance by 10 send email append comment submit form create ticket trigger deployment For non-idempotent operations, the runtime may need:
idempotency keys, compare-and-swap, transaction records, deduplication tables, state-machine transitions, or an outbox/inbox pattern. The design question is not:
Can the worker retry?
It is:
Can the external effect be safely retried?
Use state transitions instead of vague completion flags A boolean like this is often too weak:
completed = true A distributed task usually has several meaningful states.
PENDING LEASED RUNNING VERIFYING COMMITTING SUCCEEDED FAILED UNKNOWN You may also need:
CANCEL_REQUESTED CANCELLED LEASE_EXPIRED RETRYABLE_FAILURE PERMANENT_FAILURE The transitions should be explicit.
PENDING ↓ acquire LEASED ↓ start RUNNING ↓ work complete VERIFYING ↓ verifier PASS COMMITTING ↓ fenced/idempotent commit SUCCEEDED This is much easier to reason about than arbitrary worker-written flags.
Completion itself must be fenced Suppose worker A finishes after losing ownership.
It should not be allowed to mark the task SUCCEEDED.
Completion updates need the same ownership discipline as mutations.
Conceptually:
def complete_task(task_id, owner_id, epoch, result): task = load_task(task_id) if epoch < task.current_epoch: raise StaleOwner() if owner_id != task.current_owner: raise NotOwner() task.status = "SUCCEEDED" task.result = result save(task) The exact implementation will vary.
The principle does not.
Retries require a classification policy Not every failure should be retried.
A useful taxonomy is:
transient infrastructure failure rate limit worker crash network timeout stale lease invalid input verification failure policy violation permanent external error Then the retry policy can be explicit.
network timeout -> retry rate limit -> backoff + retry worker crash -> reacquire stale lease -> stop stale worker invalid input -> do not retry blindly verification failure -> revise/replan policy violation -> fail closed “Retry” is not a universal recovery mechanism.
Retries need backoff and jitter If 500 distributed workers all retry at the same moment, the recovery mechanism becomes another outage.
Use bounded backoff.
attempt 1 -> wait ~1s attempt 2 -> wait ~2s attempt 3 -> wait ~4s attempt 4 -> wait ~8s Add jitter so workers do not synchronize.
Also cap the retry budget.
max attempts max elapsed time max cost max external calls A retry policy is another budget policy.
Distributed cancellation is also not instantaneous Step 19 distinguished logical cancellation from physical cancellation.
That becomes even more important across machines.
coordinator marks CANCEL_REQUESTED ↓ message travels ↓ worker eventually observes cancellation ↓ worker stops ↓ resources released During that interval the worker may continue consuming:
tokens, CPU, GPU, tool calls, browser sessions, database connections, external API quota. The worker should therefore poll or receive cancellation signals at safe checkpoints.
But cancellation still does not authorize a stale worker to commit.
The fencing boundary remains necessary.
Ownership and cancellation should be visible in the trajectory The Step 13 observability model should now include distributed events.
lease_acquired lease_renewed lease_expired lease_reassigned heartbeat_sent heartbeat_missed worker_started worker_retried worker_cancel_requested worker_cancel_acknowledged fence_rejected idempotency_hit commit_started commit_completed Each event should include:
task_id operation_id attempt_id worker_id lease_epoch state_id policy_version resource_target cost outcome That lets you distinguish:
model failure vs worker failure vs ownership race vs retry duplication vs stale commit vs external service failure A distributed-agent runtime should think in ownership domains Not every action needs a global lock.
That would destroy concurrency.
Instead define the smallest useful ownership domain.
Examples:
repository branch customer ticket browser session dataset partition deployment target research claim search-tree node If two workers can safely operate on independent domains, they should.
The rule is:
Coordinate only where effects can conflict.
This preserves throughput while protecting consistency.
Coding agents A distributed coding system may run several repair workers.
A good isolation structure might be:
task ├── worktree A ├── worktree B └── worktree C Each worker can freely:
edit, run tests, inspect code, build artifacts. But the shared mutation boundary is the repository branch or PR.
Commit/application should require:
current task lease + fencing epoch + base commit still valid + verification PASS Before applying a result:
re-read current branch head compare with speculative base revalidate patch run required tests commit with current ownership token A stale worker should never be able to overwrite a newer accepted repair.
Research agents Distributed research workers are often safer because much of the work is read-only.
Workers can independently retrieve:
papers, webpages, datasets, quotes, counterexamples, historical context. But shared evidence stores still need deduplication and provenance.
Two workers may discover the same source.
That should not create two independent evidence votes.
Use source identity and claim identity.
source_id claim_id retrieval_attempt_id worker_id Idempotent evidence insertion is better than counting duplicate retrievals as stronger evidence.
Browser agents Browser automation can be dangerous under retries.
Read-only work is usually easy to parallelize.
open page inspect DOM collect options compare prices Mutation is harder.
submit form place order send message book reservation Those actions should have strong operation identities.
For example:
operation_id = reservation(user, restaurant, date, party_size) Before retrying, inspect whether the external state already reflects the requested action.
A timeout does not necessarily mean the action failed.
This is one of the most common distributed-agent mistakes:
request timed out ↓ assume failure ↓ retry mutation ↓ duplicate external effect A timeout means:
Outcome unknown.
That should trigger state observation before blind retry.
Data agents Distributed data agents often process partitions.
The natural ownership domain may be:
(dataset_id, partition_id, version) Each worker should write outputs under deterministic identifiers.
output_key = hash(dataset_version, partition_id, transform_version) Retries can then detect existing outputs.
But verify that the existing artifact matches the intended transform version.
Idempotency without version binding can silently reuse stale results.
DevOps agents Distributed DevOps agents need the strongest boundaries.
Suppose two workers both decide to restart a service.
Or two remediation workers both scale a cluster.
The outcome may be worse than the original incident.
Use explicit operation identities and fencing at the control-plane boundary.
incident_id service_id remediation_action resource_version lease_epoch Before executing:
verify current incident state verify current resource version verify ownership execute idempotent desired-state mutation verify postcondition Desired-state APIs are especially useful.
set replicas = 8 is usually safer than:
add 3 replicas because retries are easier to reason about.
Queue delivery and task ownership are not the same thing A queue may make a message temporarily invisible after a worker receives it.
That is useful.
It is not necessarily sufficient for distributed correctness.
A worker can retain credentials after visibility expires.
A message can be redelivered.
External mutations can outlive queue ownership.
So treat queue visibility as one ownership signal, not universal fencing.
queue visibility + lease/epoch + idempotent mutation boundary + verification is much stronger.
The outbox pattern prevents lost side effects A classic failure looks like this:
1. update database: task succeeded 2. crash 3. send completion event never happens Or the reverse:
1. send completion event 2. crash 3. database update never happens Now external observers disagree about task state.
One solution is an outbox.
Inside the same transaction that updates task state, write an event record.
transaction: update task status insert outbox event commit A separate publisher sends outbox events and marks them delivered.
Delivery itself may be at least once, so downstream consumers must deduplicate.
This is boring distributed-systems engineering.
That is exactly why advanced agent runtimes need it.
The inbox pattern deduplicates incoming work If consumers may receive the same event more than once, record processed message IDs.
Conceptually:
receive message M ↓ was M already processed? ├─ yes -> return stored result └─ no -> process + record atomically Again:
At-least-once delivery plus idempotent consumption is often more realistic than pretending duplicates never occur.
Crash recovery needs durable checkpoints Long-running agent tasks may survive multiple workers.
Do not rely on one worker’s in-memory conversation as the source of truth.
Persist meaningful checkpoints.
For example:
task state accepted plan version completed steps current environment version verified evidence memory references budget consumed pending operations A replacement worker should be able to reconstruct the task from durable state.
This connects directly to the earlier distinction:
runtime state ≠ chat history Distributed execution makes that distinction mandatory.
Checkpoints must be versioned A checkpoint should identify the exact environment it describes.
repo_commit browser_session_version dataset_version deployment_revision policy_version model_version Otherwise a new worker may resume from logically stale state.
Before continuing:
load checkpoint ↓ observe current external state ↓ compare versions ↓ resume / replan / invalidate Do not let workers invent ownership from memory This is worth making explicit.
A model may say:
I am still responsible for this task.
That is irrelevant.
Ownership is a runtime fact.
model memory ≠ lease state Workers should obtain ownership from an authoritative coordination store.
This follows the source-of-truth rule from the uncertainty and EVI posts.
Exact state should be observed directly when possible.
Distributed agents need reentrancy A task handler should tolerate being entered more than once.
That means:
reconstruct state from durable records, detect already-completed operations, verify external state, reuse idempotency keys, avoid assuming local memory is unique, and make retries explicit. A useful question is:
If this worker crashes after any line, what happens when another worker starts again?
If the answer is “we hope it doesn’t happen,” the design is not ready for distributed execution.
Side effects need a commit protocol For consequential tasks, separate preparation from commit.
prepare ↓ verify ownership ↓ verify current state ↓ verify candidate ↓ reserve idempotency key ↓ commit side effect ↓ verify postcondition This resembles Step 19’s speculative/commit boundary.
The difference is that now the boundary must survive:
worker crashes, network partitions, reassignment, duplicate attempts, delayed messages. A compact distributed task schema A practical task record might include:
from dataclasses import dataclass from typing import Optional @dataclass class DistributedTask: task_id: str status: str owner_id: Optional[str] lease_epoch: int lease_expires_at: Optional[float] operation_id: str attempt_count: int state_version: str policy_version: str result_ref: Optional[str] = None verification_status: str = "UNKNOWN" This is not enough for every production system.
It is enough to expose the important concepts.
Worker loop from first principles Conceptually:
async def worker_loop(worker_id, store, executor): while True: lease = await store.acquire_next(worker_id) if lease is None: await sleep_briefly() continue try: task = await store.load(lease.task_id) state = await executor.observe_current_state(task) if await executor.already_completed(task, state): await store.complete_if_owner( task_id=task.task_id, owner_id=worker_id, epoch=lease.epoch, result=state.existing_result, ) continue result = await executor.run(task, lease) verification = await executor.verify(task, result) if verification.status != "PASS": await store.fail_or_retry_if_owner( task.task_id, worker_id, lease.epoch, verification, ) continue await executor.commit_with_fence( task=task, result=result, epoch=lease.epoch, operation_id=task.operation_id, ) await store.complete_if_owner( task.task_id, worker_id, lease.epoch, result, ) except StaleOwner: # Another worker now owns the task. continue The key idea is not the syntax.
It is that ownership is checked repeatedly at boundaries that matter.
Measure distributed coordination separately from agent quality If a run fails, do not automatically blame the model.
Useful metrics include:
lease acquisition latency lease-expiry rate heartbeat miss rate reassignment rate duplicate-attempt rate stale-owner rejection rate idempotency-hit rate duplicate-side-effect rate retry success rate retry amplification post-cancel compute checkpoint-resume success commit conflict rate UNKNOWN-after-timeout rate Then separately measure:
verified task success false success cost per verified success critical-path latency This lets you answer:
Is the agent bad, or is the distributed runtime bad?
Duplicate work is not always wrong Step 19 deliberately launched duplicate speculative work when it reduced latency or increased candidate diversity.
Distributed duplication can therefore be intentional.
The key distinction is:
intentional duplicate computation ≠ duplicate side effect Two workers may both investigate a bug.
Only one should merge the winning fix.
Two workers may retrieve the same source.
The evidence store should deduplicate the source.
Two workers may simulate a deployment plan.
Only the selected plan should mutate production.
The distributed runtime must know which duplication is allowed and where exclusivity begins.
Ownership can be hierarchical Large agent systems may have nested work.
run ├── planning task ├── research task │ ├── source A │ └── source B └── implementation task ├── worktree A └── worktree B You may not need one global lease.
Each node can have independent ownership.
But parent cancellation and budget policies must propagate.
Useful relationships include:
parent_task_id root_run_id operation_id lease_epoch attempt_id This makes distributed traces reconstructable as graphs.
Fencing must reach the real side-effect boundary A common mistake is to validate fencing only in the coordinator database.
Example:
coordinator accepts worker B as owner worker A still has cloud credentials worker A calls cloud API directly The stale worker bypassed the fence.
When possible, enforce fencing where the side effect happens.
Examples:
compare resource version before mutation, pass generation/epoch to transactional store, require current deployment revision, use conditional writes, route privileged mutations through a fenced gateway. If the underlying API cannot enforce fencing, the risk should be explicit.
Time is not a perfect coordination primitive Lease expiry uses time.
Do not assume clocks are perfectly synchronized.
Prefer the coordination store’s authoritative lease state over each worker’s local interpretation of wall-clock time.
Also include renewal margins.
A worker should not begin a long irreversible commit when the lease is about to expire.
For example:
remaining lease: 500 ms estimated commit: 8 s The correct action is probably:
renew first or stop.
Lease duration is a policy parameter Too short:
healthy workers lose ownership, retries increase, duplicate computation rises. Too long:
recovery from dead workers is slow, tasks remain unavailable longer. Measure the distribution of task step durations and heartbeat delays.
Then choose a lease policy based on evidence.
You may use different lease durations by task class.
read-only retrieval: short long test suite: longer deployment commit: explicit protected window Again:
One global timeout is rarely a good architecture.
Retry amplification can destroy a system Suppose a dependency slows down.
Workers time out.
Each task retries.
Concurrency doubles.
The dependency becomes slower.
More tasks retry.
Now the recovery loop amplifies the outage.
slow dependency ↓ timeouts ↓ retries ↓ more load ↓ slower dependency Protect the runtime with:
concurrency limits, retry budgets, backoff, jitter, circuit breakers, queue backpressure, and cancellation. This connects directly to Step 16’s dynamic budget scheduler.
Retries consume the same finite resource envelope as reasoning and verification.
Preserve verification under retries A dangerous failure mode is:
attempt 1 runs full verification attempt 2 is a retry path attempt 2 skips verification to save time attempt 2 commits Retry paths must not silently weaken acceptance criteria.
The protected verification reserve from Step 16 still applies.
If anything, distributed ambiguity makes verification more important.
UNKNOWN is essential after ambiguous side effects Suppose a deployment request times out.
The runtime does not know whether production accepted it.
Do not classify that as FAIL immediately.
Use:
UNKNOWN Then observe current state.
timeout ↓ UNKNOWN ↓ inspect deployment state ↓ PASS / FAIL / still UNKNOWN This is a direct application of the verification and uncertainty posts.
Blind retry can be worse than waiting for evidence.
Distributed correctness can simplify agent behavior Once ownership, retries, state, and commit semantics are explicit, the model has fewer things to reason about.
That is good.
The model should not decide:
whether it still owns the task, whether a duplicate operation already happened, whether retry budgets remain, whether a stale worker may commit, whether a message was already processed. Those are runtime responsibilities.
The model can focus on the domain problem.
This follows the same principle used throughout the series:
Move deterministic control out of the language model whenever the runtime can represent it directly.
A distributed agent architecture Putting the pieces together:
task queue | coordinator | lease / epoch store / | \ worker A worker B worker C | | | isolated work / observations \ | / \ | / candidate/evidence store | selector/verifier | fenced commit gateway | external system | postcondition check The control plane owns:
task assignment, leases, epochs, retries, cancellation, budgets, policy versions. Workers own temporary execution.
The commit gateway owns consequential mutation.
The verifier owns acceptance evidence.
That separation is powerful.
Failure taxonomy Useful distributed-agent failure labels include:
LEASE_EXPIRED STALE_OWNER DUPLICATE_ATTEMPT DUPLICATE_SIDE_EFFECT IDEMPOTENCY_MISS FENCE_BYPASS HEARTBEAT_FAILURE RETRY_EXHAUSTED RETRY_AMPLIFICATION CHECKPOINT_STALE CHECKPOINT_CORRUPT COMMIT_CONFLICT CANCEL_NOT_OBSERVED POST_CANCEL_WORK AMBIGUOUS_SIDE_EFFECT OWNERSHIP_SPLIT Do not collapse all of these into:
agent error Each implies a different remediation.
Failure injection Distributed correctness should be tested deliberately.
Inject failures such as:
Worker crashes after external mutation but before completion acknowledgement Expected:
retry detects existing effect, no duplicate effect, task eventually reconciles. Worker pauses until lease expires, then resumes Expected:
new owner gets higher epoch, stale worker’s commit is rejected. Completion event delivered twice Expected:
consumer deduplicates. Network timeout after mutation request Expected:
outcome becomes UNKNOWN, runtime observes authoritative state before retry. Coordinator sends cancellation during a long model call Expected:
logical cancellation recorded, physical stop attempted, post-cancel cost measured, stale result cannot commit. Checkpoint references old environment version Expected:
resume detects mismatch, task replans or invalidates stale state. These tests are often more valuable than another hundred happy-path agent benchmarks.
Benchmark the distributed runtime fairly Compare:
single-worker baseline fixed worker pool lease-based worker pool distributed speculative pool adaptive worker pool Under the same task set.
Measure:
verified success false success duplicate side effects cost p50/p95 latency recovery time retry count stale-owner rejection idempotency hit rate The distributed system should earn its complexity.
If one worker already handles the throughput and latency requirements, do not build a miniature distributed-systems research project for aesthetics.
When do you actually need distributed agent execution? You probably do not need it when:
tasks are short, concurrency is low, one machine has enough capacity, failures can simply restart the whole run, external side effects are rare, queue delay is acceptable. You may need it when:
many independent tasks must run concurrently, model/tool workloads exceed one host, tasks are long-running, workers fail independently, specialized hardware is distributed, jobs must survive process restarts, speculative branches need many executors, throughput or latency requires horizontal scaling. The rule remains:
Add distributed coordination because measured workload requires it, not because distributed agents sound more advanced.
Ten rules for distributed agents Treat ownership as explicit runtime state. Use leases for temporary ownership, not eternal locks. Use fencing tokens to reject stale owners at mutation boundaries. Design for retries and duplicate delivery. Make side effects idempotent whenever possible. Separate logical operation identity from physical attempt identity. Persist checkpoints; do not trust one worker’s memory. Use UNKNOWN for ambiguous external outcomes. Keep verification and safety requirements intact across retries. Measure distributed coordination failures separately from model failures. The deeper architectural lesson Advanced agents eventually stop looking like a prompt around a model.
They start looking like serious software systems.
They need:
scheduling, state machines, ownership, retries, observability, transactions, isolation, verification, recovery, versioning. That is not accidental.
The more autonomy the system receives, the more important ordinary systems engineering becomes.
The final control loop now looks like this:
request ↓ policy + budget ↓ distributed task graph ↓ leases + ownership ↓ workers / speculative branches ↓ evidence + candidates ↓ selection ↓ verification ↓ fenced idempotent commit ↓ postcondition verification ↓ trajectory + durable checkpoint The model is still important.
But the model is no longer the architecture.
What comes next? Distributed execution creates another problem.
Now your system may have:
hundreds of workers, many concurrent task graphs, shared model capacity, rate-limited tools, GPU pools, browser pools, database connections, and multiple tenants competing for the same resources. Per-task budgeting is no longer enough.
The next question becomes:
How should an agent platform schedule many competing runs fairly under shared capacity constraints?
That means moving from per-agent budgets to global admission control, quotas, priorities, fairness and backpressure.
That is the next stage.