Reliability on Programmer.ie: Modern AI programming

Build a Production AI Agent From First Principles: The Complete Reference Architecture

Sun, 09 Aug 2026 16:09:00 +0100

Build a Production AI Agent From First Principles: The Complete Reference Architecture

We have spent this series adding mechanisms only when a specific failure demanded them.

We started with a model call.

Then we added candidate generation, critique, planning, tool use, memory, search and verification.

Then the system stopped looking like a clever prompt.

It started looking like software.

Then distributed systems problems arrived:

duplicate work,
retries,
leases,
fencing,
backpressure,
dependency failure,
behavioral drift,
release compatibility,
replay,
incident forensics,
SLOs,
authority,
competence,
capability acquisition,
placement,
handoff,
stale state,
stale intent,
commitments,
durable workflows,
transaction recovery,
trust boundaries,
multi-agent coordination,
and finally an explicit control plane.

At this point the architecture is complete enough that adding another isolated mechanism would make the series worse rather than better.

Advanced Agents From First Principles 24: How Do You Release Agent Behavior Safely? Add Behavioral Contracts, Compatibility Checks and Promotion Gates

Sun, 09 Aug 2026 12:09:00 +0100

A model change can pass health checks and still break your agent.

A prompt change can preserve output schema and still change which tools the agent chooses.

A router change can improve average cost and silently starve a specialist that is essential for one high-risk workload.

A memory migration can keep every row intact and still change which memories are retrieved.

A verifier upgrade can make every dashboard greener while making the system less trustworthy.

Advanced Agents From First Principles 23: Your Infrastructure Is Healthy. Why Is the Agent Getting Worse? Detect Behavioral Drift and Roll Back Safely

Sun, 09 Aug 2026 12:00:00 +0100

Your Infrastructure Is Healthy. Why Is the Agent Getting Worse?

Your dashboards are green.

The model endpoint is responding.

The browser workers are alive.

The database is healthy.

The queue is draining.

The verifier service is up.

Latency has not exploded.

There are no obvious exceptions.

And yet the agent is getting worse.

It fixes fewer bugs.

It retrieves weaker evidence.

It escalates to expensive models more often.

It chooses the wrong tools more frequently.

Advanced Agents From First Principles 22: What Happens When One Dependency Starts Failing? Add Circuit Breakers, Bulkheads and Graceful Degradation

Sun, 09 Aug 2026 11:26:00 +0100

What Happens When One Dependency Starts Failing?

An advanced agent platform can be working perfectly and still collapse.

Not because the planner became confused.

Not because the router chose the wrong model.

Not because MCTS explored the wrong branch.

Because one dependency became unhealthy.

Maybe:

the frontier model starts returning 503s,
the browser pool starts timing out,
the embedding service slows from 80 ms to 8 seconds,
the verifier database starts dropping connections,
the search API starts rate limiting,
a sandbox cluster stops accepting jobs,
or an external provider is technically up but so slow that every request consumes a worker for minutes.

Then the retries begin.

Advanced Agents From First Principles 20: Can Your Agent Coordinate Across Machines Without Duplicating Work? Use Leases, Idempotency and Fencing

Sun, 09 Aug 2026 11:13:00 +0100

Can Your Agent Coordinate Across Machines Without Duplicating Work?

A single-process agent can already be complicated.

It can plan.

It can search.

It can launch speculative branches.

It can cancel losing work.

It can verify outcomes.

Then you move that work onto multiple workers.

Now a new class of failure appears.

Two workers both believe they own the same task.

One worker pauses for thirty seconds.

Another worker assumes it died and takes over.

Advanced Agents From First Principles 25: Can You Reproduce an Agent Run Months Later? Add Deterministic Replay and Provenance

Sun, 09 Aug 2026 00:00:00 +0000

A production agent fails on Tuesday.

You inspect it on Wednesday and cannot reproduce the failure.

Three weeks later someone asks a harder question:

What exactly did this agent see, which release was running, which tools were called, which evidence was used, and why did the verifier accept the result?

If the answer is “we have some logs”, you do not yet have reproducibility.

You have fragments.

Advanced agents accumulate hidden variability quickly:

Advanced Agents From First Principles 33: Which Shared Components Actually Unlock More Capability? Build a Capability Dependency Graph

Sun, 09 Aug 2026 00:00:00 +0000

A practical capability-dependency architecture for advanced agents: identify shared primitives that unlock many capabilities, quantify leverage, expose correlated-failure hotspots, and invest in platform components without creating hidden systemic risk.

Advanced Agents From First Principles 34: Where Should This Task Actually Run? Build Capability-Aware Placement Across Models, Providers and Resource Pools

Sun, 09 Aug 2026 00:00:00 +0000

A practical placement architecture for advanced agents: route work across local and frontier models, providers, regions, GPUs, browser pools and specialist runtimes using demonstrated competence, verifier availability, policy constraints, health, cost and latency rather than model prestige.

Advanced Agents From First Principles 38: A Plan Is Not a Commitment — Model Goals, Commitments and Executable Work

Sun, 09 Aug 2026 00:00:00 +0000

A practical architecture for separating goals, plans, commitments, tasks and actions in long-running agents so replanning, cancellation, handoff and external obligations remain correct.

Advanced Agents From First Principles 40: Your Agent Changed the World. What Happens When Step Two Fails? Build Transactions, Compensation and Reconciliation

Sun, 09 Aug 2026 00:00:00 +0000

A practical architecture for agent workflows that span systems without a global transaction: classify side effects, prepare carefully, commit with identity, verify externally, compensate when possible, reconcile ambiguity, and never pretend rollback is free.