# Advanced Agents From First Principles 40: Your Agent Changed the World. What Happens When Step Two Fails? Build Transactions, Compensation and Reconciliation
# Book: Advanced Agents From First Principles
# Chapter URL: https://programmer.ie/books/advanced-agents-from-first-principles/40-chapter/
# Prose fingerprint (sha256, normalized): 93fa35e5b9143fcf6fad36934db735898114874c14e60d667ffb2e3c9f136be8
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

An agent can reason correctly, choose the right tool, pass authorization, and still leave the world in a broken intermediate state.
The failure does not require a hallucination.
It can happen because step one succeeds and step two does not.
create deployment ✓ update DNS ✓ run migration ✓ start application ✗ notify customer not attempted Now what?
A naive agent architecture often has only two ideas:
success failure But the external world may already have changed.
There may be no global rollback button.
There may not even be one authoritative system that knows whether the last request succeeded.
This is where agent engineering stops looking like prompt engineering and starts looking like distributed transaction engineering.
The core principle for this chapter is:
Rollback is not the inverse of every action.
The correct runtime needs explicit semantics for preparation, commitment, verification, compensation and reconciliation.
The Search Problem: “How Do I Roll Back an AI Agent After a Partial Failure?” Consider an agent that has to onboard a customer.
It might need to:
create a tenant in the application database, create a billing customer, provision cloud resources, create DNS records, configure access control, send a welcome email. Those actions may span six independent systems.
There is no database transaction wrapping all of them.
If step four fails, the system must answer several different questions:
What definitely happened? What may have happened? What can be undone? What can only be compensated? What is irreversible? What commitments are still active? What state must be reconciled before retrying? That is not one error-handling branch.
It is a transaction model.
Why ACID Does Not Magically Extend Across Tools Inside one relational database we may have:
BEGIN UPDATE ... INSERT ... COMMIT The database coordinates atomicity.
An agent workflow commonly crosses boundaries like:
GitHub Stripe AWS Kubernetes email provider browser session internal database human approval queue Those systems do not share one transaction coordinator.
Even if two systems each offer transactions internally, their transactions are not automatically atomic together.
The agent therefore needs to reason about a sequence of durable external effects.
But the model should not invent the transaction semantics.
Those semantics belong in the execution platform.
First Principle: Classify Side Effects Before You Execute Them Not every action has the same recovery properties.
A useful minimum classification is:
REVERSIBLE COMPENSATABLE IRREVERSIBLE UNKNOWN Reversible A reversible operation has a trusted inverse that restores the relevant prior state closely enough for the system’s correctness requirements.
Example:
set feature flag ON may be reversible by:
set feature flag OFF provided no irreversible downstream effects occurred while the flag was enabled.
That last condition matters.
The inverse command existing does not prove the whole effect is reversible.
Compensatable A compensatable action cannot literally be erased, but another action can reduce or neutralize its business effect.
Example:
charge card may be compensated by:
issue refund The charge still occurred.
The refund is a second real-world event.
Accounting records, notifications and timing all remain part of history.
So:
refund != undo(charge) Irreversible Some effects cannot reliably be undone.
Examples may include:
send email publish public message delete unrecoverable external data trigger irreversible physical action reveal secret to recipient Once executed, the correct recovery action may be explanation, escalation or mitigation rather than rollback.
Unknown This is the most dangerous category.
Suppose the agent calls an external API and receives a timeout.
POST /create-order ↓ network timeout The timeout does not prove failure.
The remote system may have committed the order and lost the response.
The operation is now:
OUTCOME_UNKNOWN Retrying blindly can duplicate the side effect.
A Production Transaction Lifecycle For consequential external effects, use an explicit lifecycle:
prepare ↓ validate ↓ commit ↓ observe authoritative result ↓ verify postcondition ↓ complete And on failure:
failure / ambiguity ↓ classify current reality ↓ compensate if appropriate ↓ verify compensation ↓ reconcile remaining state ↓ complete / escalate / UNKNOWN The important detail is that recovery has verification too.
A compensation request is not proof that compensation succeeded.
Prepare Before Commit Many failures are cheaper to prevent than repair.
The prepare phase should gather everything that can be checked before mutation.
For example:
intent current? authority valid? state fresh? competence sufficient? verifier available? dependent resources available? idempotency key assigned? operation identity assigned? compensation plan known? commitments understood? A simplified DTO might look like:
from dataclasses import dataclass from enum import Enum class RecoveryClass(str, Enum): REVERSIBLE = "reversible" COMPENSATABLE = "compensatable" IRREVERSIBLE = "irreversible" UNKNOWN = "unknown" @dataclass(frozen=True) class PreparedOperation: operation_id: str intent_id: str intent_version: int action_type: str target_ref: str state_version: str authority_ref: str idempotency_key: str recovery_class: RecoveryClass compensation_operation_type: str | None expected_postcondition: str The important property is not this exact Python shape.
The important property is that recovery semantics are known before a consequential action is allowed to commit.
Commit With Stable Operation Identity Every consequential logical operation should have a stable identity.
logical operation ↓ operation_id = OP-123 ↓ physical attempt 1 physical attempt 2 physical attempt 3 Do not assign a new business operation identity just because a network retry occurred.
That enables idempotency.
For example:
operation_id: OP-123 idempotency_key: customer-827:create-billing-account:v1 If the provider supports native idempotency keys, use them.
If it supports conditional writes, use them.
If it supports compare-and-set or version preconditions, use them.
The model should not simulate guarantees the external system can enforce exactly.
Exactly Once Is Usually the Wrong Promise Distributed systems make exactly-once side effects difficult.
A more useful design is:
at-least-once delivery + idempotent logical operation + authoritative reconciliation Suppose the worker sends:
create invoice INV-927 and crashes after the remote service commits it but before recording success.
On restart, the platform should not ask the model:
“Do you think the invoice probably exists?”
It should query authoritative state using the stable operation identity.
lookup INV-927 Then transition based on reality.
Postcondition Verification Is Part of the Transaction An API returning 200 OK is not always the business outcome you intended.
The agent may request:
deploy version 42 and receive success from the deployment API.
The actual postcondition may be:
version 42 serving traffic health checks passing error rate below threshold schema compatible So the mutation pipeline should be:
request accepted ↓ external mutation ↓ read authoritative state ↓ verify desired postcondition The verifier is external to the action generator.
A tool response saying success=true is evidence about the request, not necessarily proof of the intended outcome.
Compensation Is a New Transaction This is one of the most important rules in the chapter.
Compensation is not metadata attached to the failed action. It is another consequential workflow with its own authority, identity, retries and verification.
If the original operation was:
charge €100 and the compensation is:
refund €100 then the refund should have its own:
operation_id idempotency_key authority decision state preconditions external result postcondition verification provenance A minimal record might be:
@dataclass(frozen=True) class CompensationRequest: compensation_id: str original_operation_id: str compensation_type: str reason: str expected_postcondition: str The connection to the original operation is explicit.
But the compensation is not treated as if the original operation never happened.
Compensation Can Fail Too Imagine:
reserve inventory ✓ charge payment ✓ create shipment ✗ The workflow decides to compensate:
release inventory ✓ refund payment ✗ Now the system is still inconsistent.
The correct state is not:
FAILED It is more specific:
COMPENSATION_FAILED RECONCILIATION_REQUIRED This matters operationally because cleanup work must remain visible and durable.
It must not disappear because the original customer-facing workflow has already failed.
Reconciliation Is Different From Compensation Compensation asks:
What corrective action should we take?
Reconciliation asks:
What is actually true now?
That distinction is crucial after ambiguous outcomes.
Suppose:
create DNS record ↓ timeout Before deciding whether to retry or compensate, reconcile:
read authoritative DNS state ↓ record exists? yes / no / unknown record has expected value? yes / no / unknown Then choose the next transition.
Reconciliation is therefore often:
observe first act second rather than:
failed request → immediately issue opposite request Use a Transaction Ledger For multi-system workflows, persist a transaction ledger.
For example:
@dataclass class OperationRecord: operation_id: str workflow_id: str intent_id: str intent_version: int action_type: str target_ref: str recovery_class: RecoveryClass state: str idempotency_key: str attempt_count: int external_reference: str | None compensation_operation_id: str | None verification_ref: str | None Possible states:
PREPARED COMMITTING COMMITTED VERIFYING VERIFIED OUTCOME_UNKNOWN COMPENSATION_REQUIRED COMPENSATING COMPENSATED COMPENSATION_FAILED RECONCILIATION_REQUIRED RECONCILED IRREVERSIBLE_EFFECT Do not compress these into one boolean called success.
Connect the Ledger to the Durable Workflow Step 39 gave us a durable workflow runtime.
The operation ledger fits inside it naturally:
workflow │ ├── operation A ── VERIFIED │ ├── operation B ── VERIFIED │ ├── operation C ── OUTCOME_UNKNOWN │ │ │ └── reconciliation activity │ └── protected cleanup path The workflow can sleep, restart, migrate workers or wait for a provider without losing the transaction state.
The model does not need to remember which effects occurred.
That is durable control-plane data.
Side-Effect Graphs Are Better Than Flat Lists Real workflows are often not simple sequences.
Imagine:
create tenant ├── create billing account ├── create storage └── create DNS ↓ deploy service The correct compensation order may depend on this graph.
For example, you may need to stop traffic before deleting storage.
So model side-effect dependencies explicitly:
operation A ENABLES B operation B DEPENDS_ON A operation C MUST_COMPENSATE_BEFORE A This lets the workflow calculate a safe recovery order.
Do not assume reverse chronological order is always correct.
Saga-Like Semantics Without Pretending Everything Is Reversible A useful conceptual model is a saga:
T1 → T2 → T3 → T4 with compensations:
C1 ← C2 ← C3 But there are two important corrections for agent systems.
First, not every transaction step has a valid compensation.
Second, the compensation order may depend on current external state rather than merely being a predefined reverse list.
So the agent platform should represent recovery explicitly rather than assuming:
compensate(step_n ... step_1) will always restore correctness.
Irreversible Effects Change Planning If an action is irreversible, that information should affect the plan before execution.
For example:
prepare draft email verify recipient verify content human approval if required send email The irreversible step should move toward the end of the workflow whenever practical.
This is a general design principle:
Delay irreversible side effects until uncertainty has been reduced as far as economically reasonable.
That connects directly to earlier chapters on verification, uncertainty and Expected Value of Information.
Before an irreversible action, buying one more authoritative observation may have unusually high value.
Order Work by Recoverability Suppose two plans produce the same successful outcome.
Plan A:
irreversible action reversible action verification Plan B:
verification reversible action irreversible action postcondition verification All else equal, Plan B has a better recovery profile.
A planner can therefore consider recoverability as a constraint or decision feature.
Do not let the model invent numeric precision here.
A deterministic risk class is often enough:
LOW_RECOVERY_RISK MODERATE_RECOVERY_RISK HIGH_RECOVERY_RISK IRREVERSIBLE Commitments Survive Partial Failure Step 38 introduced commitments.
Transactions interact with them directly.
Suppose a workflow reserves a deployment window and creates a maintenance notification.
Then deployment fails.
The failed workflow cannot simply terminate.
It must ask:
Is the maintenance window still active? Can it be released? Was the customer notified? Does a follow-up commitment now exist? The transaction recovery path therefore includes commitment resolution:
satisfy release transfer compensate reconcile Cancellation Does Not Erase Transactions Step 37 established that intent can be cancelled or superseded.
Suppose cancellation arrives after two of five side effects have committed.
intent v7 ACTIVE operation A VERIFIED operation B COMMITTED operation C not started cancel intent v7 Correct behavior:
fence future business mutations stop C, D, E inspect A and B resolve commitments compensate or reconcile if policy requires verify final state Incorrect behavior:
cancel workflow forget everything Cancellation may create more mandatory work, not less.
Cleanup and reconciliation need protected capacity.
Compensation Must Respect Current Intent Too There is a subtle problem.
Suppose intent v7 is cancelled and a compensation is scheduled.
Then intent v8 arrives and explicitly wants to preserve one of the effects created by v7.
Blindly executing the old compensation could now be wrong.
So compensation should be bound to:
original operation current recovery policy current intent lineage current external state Before committing compensation, revalidate whether the compensation is still appropriate.
This is the same temporal-consistency principle from Step 36 applied to recovery work.
Human Approval Can Be Part of Recovery Some recovery decisions should not be autonomous.
For example:
customer charged inventory committed shipment creation failed The allowed recovery choices may be:
refund customer hold order for manual fulfillment substitute inventory source escalate to support Those are business decisions, not merely technical inverses.
The workflow can prepare evidence and options while authority remains with a human.
Remember the Step 29 distinction:
authorization ≠ verification A human can authorize a refund.
The platform must still verify that the refund actually occurred.
Ambiguous Effects Should Block Unsafe Follow-Up If the state of an operation is unknown, downstream work may need to stop.
For example:
payment status UNKNOWN should usually prevent:
ship order until reconciliation establishes whether payment succeeded.
That means operation dependencies can carry predicates such as:
requires VERIFIED requires NOT_COMMITTED requires RECONCILED This prevents the workflow from treating uncertainty as failure or success by convenience.
UNKNOWN Is Not a Temporary Embarrassment The platform should be comfortable ending a decision cycle with:
OUTCOME_UNKNOWN RECONCILIATION_REQUIRED HUMAN_REQUIRED if authoritative reality cannot currently be established.
The dangerous alternative is fabricating certainty so the workflow can continue.
Example: GitHub Deployment Workflow Consider an agent that prepares and deploys a code change.
1. create branch 2. commit code 3. open PR 4. merge PR 5. deploy release 6. verify production Recovery classes may be:
create branch reversible commit code reversible-ish / preserved history open PR compensatable by close merge PR not cleanly reversible production deploy compensatable by new rollback release external message potentially irreversible If production verification fails after merge, the agent should not pretend the Git history can simply be erased.
The recovery may be:
create corrective commit open rollback PR verify rollback redeploy verify production record incident That is compensation and forward recovery, not literal rollback.
Example: Browser Purchase A browser agent might:
select item enter delivery address submit payment receive timeout The timeout creates an ambiguous side effect.
Do not:
click Buy again Instead:
query order history query payment state match stable transaction identifiers reconcile Only then determine whether a retry is valid.
This is why browser automation needs transaction semantics just as much as backend APIs.
Example: Infrastructure Automation An infrastructure agent might:
create database create secrets create service attach load balancer update DNS If DNS update fails, deleting everything may be worse than finishing the deployment later.
Recovery policy may choose:
KEEP_AND_RETRY COMPENSATE_PARTIAL ROLL_FORWARD ESCALATE The correct choice depends on commitments, risk, cost, freshness and operational policy.
Do not encode every failure as “delete what we created.”
Roll Forward Is Often Safer Than Roll Back Many systems are easier to repair by completing a corrected forward transition.
Examples:
bad database migration → corrective migration bad deployment → new known-good deployment incorrect configuration → corrected configuration version This preserves history and avoids pretending the previous world can be reconstructed exactly.
A useful recovery policy therefore includes:
ROLL_FORWARD COMPENSATE REVERSE RECONCILE HUMAN_ESCALATE NO_AUTOMATED_RECOVERY Make Recovery Policy Explicit A platform can define recovery policy separately from model reasoning.
@dataclass(frozen=True) class RecoveryPolicy: action_type: str recovery_class: RecoveryClass automatic_compensation_allowed: bool human_approval_required: bool max_compensation_attempts: int reconciliation_required_after_timeout: bool The model may suggest a recovery strategy.
It should not be able to rewrite the safety constraints governing its own failed action.
Compensation Authority Should Be Narrow A common mistake is to grant a cleanup worker broad privileges because it is “only undoing things.”
But compensation can be as consequential as the original action.
A refund changes money.
Deleting a resource destroys state.
Revoking access changes authorization.
Therefore:
original action authority and:
compensation authority should be explicit and independently scoped.
Recovery Has Its Own Budget Earlier chapters protected verification capacity.
Durable systems should also protect reconciliation and cleanup capacity.
If the platform is overloaded and sheds all work indiscriminately, it can strand half-completed workflows.
A useful resource split is:
new work speculative work verification work reconciliation work critical compensation work Under severe pressure:
shed speculation first slow new work preserve verification preserve reconciliation preserve required compensation Otherwise overload can convert temporary failures into lasting inconsistency.
Transaction State Belongs in Replay and Provenance Step 25 introduced deterministic replay and provenance.
Every consequential operation should record enough information to reconstruct:
why action was attempted which intent authorized it which state version was observed which operation identity was used which provider/system received it what response occurred what authoritative state was later observed what verifier concluded whether compensation occurred what final state remained That enables incident investigators to distinguish:
request failed response was lost remote commit succeeded verification failed compensation failed reconciliation never ran Those are radically different incidents.
Transaction Events for Trajectory Observability Useful structured events include:
operation.prepared operation.commit_started operation.response_received operation.outcome_unknown operation.postcondition_verified operation.compensation_requested operation.compensation_started operation.compensation_verified operation.reconciliation_started operation.reconciliation_completed operation.human_escalated operation.irreversible_effect_recorded Each event should carry stable identifiers rather than relying on prose logs.
Metrics That Matter Do not optimize for “few rollbacks” in isolation.
That can simply mean failures are being ignored.
Track metrics such as:
partial-commit rate ambiguous-outcome rate reconciliation latency compensation success rate compensation failure rate irreversible-effect rate duplicate-side-effect rate idempotency conflict rate postcondition failure rate recovery human-escalation rate orphaned-operation count mean time to consistent state One especially useful metric is:
time_to_consistent_state because user-facing failure may happen before the system is operationally clean again.
Recovery Quality Matters More Than Workflow Completion A workflow that reports FAILED in ten seconds but leaves three external systems inconsistent is not necessarily healthier than one that spends five minutes reconciling and reaches a known safe state.
So terminal states should distinguish:
FAILED_CLEAN FAILED_COMPENSATED FAILED_RECONCILED FAILED_WITH_IRREVERSIBLE_EFFECT FAILED_RECONCILIATION_REQUIRED UNKNOWN This makes operational reality visible.
Failure Injection Is Mandatory Transaction recovery should be tested by deliberately failing at every meaningful boundary.
For example:
crash before request crash after send before response provider commits then times out provider rejects duplicate idempotency key verification service unavailable compensation provider unavailable compensation commits then response lost intent cancelled during compensation state changes before compensation worker loses lease during reconciliation human approval expires during recovery The invariant is not:
workflow never fails It is:
Every consequential side effect reaches a known verified state, a protected recovery path, or an explicit unresolved state that cannot silently progress.
Deterministic Transaction Invariants Some checks should never depend on an LLM.
For example:
def may_execute_downstream(parent: OperationRecord) -> bool: return parent.state in {"VERIFIED", "RECONCILED"} def may_retry_unknown(operation: OperationRecord) -> bool: return operation.state != "OUTCOME_UNKNOWN" In reality, retryability can be more nuanced.
But the principle stands:
Known exact state-machine facts belong in deterministic control code.
Do You Actually Need a Transaction Layer? Not every agent does.
If the system only:
reads documents produces suggestions writes drafts and never performs consequential external mutations, a complex compensation engine may be unnecessary.
Start with the simplest boundary that matches the risk.
A useful progression is:
read-only agent ↓ typed side effects + postcondition verification ↓ idempotent operations ↓ durable operation ledger ↓ compensation / reconciliation ↓ multi-system recovery graph Add each layer because measured failure requires it.
What This Adds to the Architecture The series now has:
intent lifecycle ↓ goals and commitments ↓ durable workflow ↓ transaction semantics ↓ external side effects The durable workflow answers:
Where are we in the process?
The transaction layer answers:
What has actually changed in the world, and what must happen if only part of the intended change succeeded?
That is a different question.
Both are necessary for consequential autonomy.
The Deeper Principle Agent systems are often described as systems that can “take actions.”
That description is incomplete.
Production systems need to know:
which action was intended which action was authorized which action was attempted which action definitely happened which action may have happened which outcome was verified which obligation remains which recovery is possible which recovery was attempted which final state is now true That is the difference between action generation and operational control.
The final principle for this step is:
Do not design agents around successful action sequences. Design them around partial success, ambiguity and recovery.
The next stage is trust.
A system can now reason, coordinate, survive failure, reconcile external reality and recover from partial transactions.
But it still consumes instructions and evidence from users, tools, retrieved documents, web pages, other agents and generated code.
Those sources do not deserve equal authority.
The next question is therefore:
What should your agent trust?