What Happens When One Dependency Starts Failing? Add Circuit Breakers, Bulkheads and Graceful Degradation
What Happens When One Dependency Starts Failing?
An advanced agent platform can be working perfectly and still collapse.
Not because the planner became confused.
Not because the router chose the wrong model.
Not because MCTS explored the wrong branch.
Because one dependency became unhealthy.
Maybe:
- the frontier model starts returning 503s,
- the browser pool starts timing out,
- the embedding service slows from 80 ms to 8 seconds,
- the verifier database starts dropping connections,
- the search API starts rate limiting,
- a sandbox cluster stops accepting jobs,
- or an external provider is technically up but so slow that every request consumes a worker for minutes.
Then the retries begin.
And the retries create more load.
And the extra load makes the dependency slower.
And the slower dependency causes more timeouts.
And more timeouts create more retries.
Soon a small local problem becomes a platform-wide incident.
The central rule of this article is:
A dependency failure should reduce capability locally, not destabilize the entire agent platform.
That means the runtime needs more than retries.
It needs failure containment.
healthy platform
|
v
dependency degrades
|
v
health detection
|
v
circuit breaker
|
+----> fallback / degraded mode
|
+----> isolate capacity with bulkhead
|
+----> bounded probe traffic
|
v
recovery or continued containment
This is where advanced agent engineering starts to look much more like ordinary distributed-systems engineering.
That is a good thing.
The Real Failure Is Usually the Cascade
Suppose your agent uses this pipeline:
request
↓
retrieval
↓
planner
↓
frontier model
↓
browser
↓
verifier
Now imagine the browser service becomes slow.
A naive runtime does this:
browser timeout
↓
retry
↓
retry
↓
new agent branch
↓
more browser calls
↓
more timeouts
↓
more retries
The browser failure is now affecting:
- worker occupancy,
- queue depth,
- model-call budgets,
- verifier availability,
- platform latency,
- tenant quotas,
- and possibly external API limits.
The problem is no longer “the browser is slow.”
The problem is:
the architecture allowed one dependency’s failure mode to consume resources outside its own failure domain.
This is what bulkheads and circuit breakers are designed to prevent.
1. Dependency Health Is Not Binary
A dependency is rarely just:
UP
DOWN
For an agent runtime, a more useful health model is:
HEALTHY
DEGRADED
FAILING
OPEN
RECOVERING
A provider can be technically reachable while still being operationally unusable.
Useful health signals include:
- success rate,
- timeout rate,
- p50 latency,
- p95 latency,
- p99 latency,
- rate-limit responses,
- malformed-response rate,
- verifier disagreement,
- queue age,
- resource saturation,
- and postcondition failure rate.
This matters because an LLM endpoint that returns HTTP 200 in 45 seconds may be less healthy for an interactive workload than one that returns HTTP 503 immediately.
The runtime should reason about service quality, not merely process availability.
2. Circuit Breakers Stop Repeating Known Failures
A circuit breaker protects a failing dependency from endless requests while protecting the caller from endless waiting.
A basic breaker has three states:
CLOSED
|
failures exceed threshold
v
OPEN
|
cooldown expires
v
HALF_OPEN
|
+---- success ----> CLOSED
|
+---- failure ----> OPEN
The names are electrical metaphors:
- CLOSED means requests flow normally.
- OPEN means calls are blocked immediately.
- HALF_OPEN means limited probe traffic is allowed to test recovery.
The important property is this:
When failure is already well established, the runtime stops paying to rediscover it on every request.
A Minimal Circuit Breaker
from dataclasses import dataclass
from enum import Enum
import time
class BreakerState(str, Enum):
CLOSED = "closed"
OPEN = "open"
HALF_OPEN = "half_open"
@dataclass
class CircuitBreaker:
failure_threshold: int = 5
recovery_timeout_s: float = 30.0
state: BreakerState = BreakerState.CLOSED
consecutive_failures: int = 0
opened_at: float | None = None
def allow(self) -> bool:
if self.state == BreakerState.CLOSED:
return True
if self.state == BreakerState.OPEN:
assert self.opened_at is not None
if time.monotonic() - self.opened_at >= self.recovery_timeout_s:
self.state = BreakerState.HALF_OPEN
return True
return False
return True
def record_success(self) -> None:
self.state = BreakerState.CLOSED
self.consecutive_failures = 0
self.opened_at = None
def record_failure(self) -> None:
self.consecutive_failures += 1
if self.state == BreakerState.HALF_OPEN:
self._open()
return
if self.consecutive_failures >= self.failure_threshold:
self._open()
def _open(self) -> None:
self.state = BreakerState.OPEN
self.opened_at = time.monotonic()
This implementation is deliberately simple.
Production systems often use rolling windows, weighted errors, latency thresholds, minimum sample counts and probe limits.
But the underlying decision remains understandable:
is this dependency healthy enough to justify another call?
3. A Timeout Is Not Automatically a Dependency Failure
This distinction matters.
Suppose a browser call times out because:
- the browser pool is overloaded,
- the target website is slow,
- DNS is failing,
- your own worker was paused,
- or the task deadline was shorter than normal.
These have different causes.
Do not open a global browser breaker because one unusual page took too long.
A useful breaker needs enough context to identify the actual failure domain.
For example:
provider=openai
model=gpt-x
region=eu-west
operation=chat_completion
or:
browser_pool=interactive-eu
site=example.com
operation=navigate
The breaker key should match the dependency boundary you can actually isolate.
Too broad:
browser = broken
Too narrow:
run-849-attempt-12-browser-call-4 = broken
Useful:
browser_pool=eu-west / operation=navigate
or sometimes:
target_domain=example.com
The correct scope depends on the failure.
4. Bulkheads Prevent One Failure From Consuming Everything
Circuit breakers stop calls after failure becomes obvious.
Bulkheads prevent the failure from consuming the whole platform before the breaker opens.
The term comes from ships.
A ship is divided into watertight compartments.
If one compartment floods, the whole ship does not immediately sink.
Agent platforms need the same principle.
Without bulkheads:
all workloads
↓
shared worker pool
↓
failing browser dependency
↓
all workers blocked
With bulkheads:
coding agents ──> worker pool A
research agents ─> worker pool B
browser agents ──> worker pool C
verification ────> protected pool D
Or by dependency:
frontier model semaphore: 50
browser semaphore: 20
retrieval semaphore: 40
verifier semaphore: 30
Now a browser collapse cannot consume every verifier slot.
That matters because verification may be required to safely terminate already-running tasks.
Bulkheads Are Resource Boundaries
A bulkhead can be implemented with:
- separate queues,
- separate worker pools,
- separate connection pools,
- separate semaphores,
- per-provider concurrency limits,
- per-tenant limits,
- reserved capacity,
- or physically separate infrastructure.
The important question is:
What failure should be prevented from consuming what other resource?
This is closely related to Step 21’s platform scheduler.
The scheduler decides how shared capacity is allocated.
The bulkhead defines which capacity cannot be consumed across a failure boundary.
5. Retries Must Be Contained
Retries are one of the easiest ways to turn a small failure into a large one.
Imagine:
100 active runs
× 3 retries
× 4 speculative branches
= 1,200 calls
A dependency that was already overloaded now receives twelve times the obvious workload.
This is retry amplification.
The rule should be:
Retry only when the failure is plausibly transient, the operation is safe to retry, the retry budget remains, and the platform has capacity to absorb it.
A retry policy should consider:
failure type
idempotency
attempt count
breaker state
platform pressure
time remaining
external rate limits
Retry policy should also use backoff and jitter.
import random
def retry_delay(attempt: int, base: float = 0.5, cap: float = 30.0) -> float:
exponential = min(cap, base * (2 ** attempt))
return random.uniform(0, exponential)
Jitter prevents thousands of workers from retrying simultaneously when a breaker recovers.
6. Backpressure and Circuit Breakers Solve Different Problems
These mechanisms are related but not interchangeable.
Backpressure says:
downstream capacity is constrained
→ slow or stop upstream production
A circuit breaker says:
dependency is unhealthy
→ stop attempting the operation temporarily
A healthy service can still require backpressure because demand exceeds capacity.
A low-load service can still require a circuit breaker because it is returning incorrect results.
The platform should distinguish:
capacity problem
health problem
correctness problem
Different problems require different responses.
7. Graceful Degradation Means Choosing What to Lose
A degraded platform should not simply behave like a slower healthy platform.
It should intentionally reduce capability.
Suppose a research agent normally does:
query rewrite
↓
parallel retrieval from 4 sources
↓
cross-source comparison
↓
critic
↓
secondary verifier
If one retrieval provider fails, a graceful degradation might be:
query rewrite
↓
retrieval from 2 healthy sources
↓
cross-source comparison
↓
primary verifier
If the verifier is unavailable, the correct degradation might instead be:
result produced
↓
UNKNOWN / UNVERIFIED
not:
result produced
↓
pretend PASS
That distinction is crucial.
Graceful degradation may reduce capability. It must not silently reduce truth standards.
8. Build Explicit Service Modes
Do not let degradation emerge accidentally from random timeouts.
Represent it explicitly.
from enum import Enum
class ServiceMode(str, Enum):
NORMAL = "normal"
DEGRADED = "degraded"
MINIMAL = "minimal"
EMERGENCY = "emergency"
A mode can control:
- allowed models,
- search width,
- number of critics,
- parallelism,
- retrieval providers,
- optional enrichment,
- background work,
- and fallback behavior.
For example:
NORMAL
all features
DEGRADED
fewer branches
cheaper models where safe
optional critics disabled
MINIMAL
deterministic paths
primary model only
mandatory verification preserved
EMERGENCY
critical workflows only
no speculative work
protected verification only
The platform scheduler from Step 21 can activate these modes based on global pressure.
Circuit breakers can activate dependency-specific degraded modes.
These are different triggers controlling the same runtime surface.
9. Fallbacks Must Preserve Semantics
A fallback is not useful merely because it returns something.
Suppose the primary model fails.
Fallback options include:
frontier model
↓ fail
local model
That may be acceptable for:
- summarization,
- classification,
- routing,
- or formatting.
It may be unacceptable for:
- a complex code migration,
- safety-critical analysis,
- or high-stakes verification.
So the fallback decision should depend on capability requirements, not merely availability.
A useful fallback contract contains:
required capability
required context size
required tool support
required structured-output reliability
required verifier coverage
acceptable latency
acceptable cost
If no fallback satisfies the contract, return:
UNKNOWN
DEFERRED
DEPENDENCY_UNAVAILABLE
Do not silently substitute a weaker component and pretend the original guarantee still holds.
10. Stale-But-Safe Reads Can Be Better Than Failure
Not all stale data is dangerous.
Suppose a retrieval index refresh is temporarily unavailable.
If the task is:
summarize a stable internal design document
a cached copy from an hour ago may be perfectly adequate.
If the task is:
what is the current production deployment state?
an hour-old snapshot may be dangerous.
So stale fallbacks require freshness semantics.
A useful observation record contains:
@dataclass
class Observation:
value: object
source: str
observed_at: float
state_id: str | None
freshness_class: str
And the policy asks:
is this stale observation still valid for this decision?
This connects directly to Step 17’s state uncertainty and Step 18’s Expected Value of Information.
Sometimes the best action under dependency failure is:
use cached evidence
Sometimes it is:
return UNKNOWN
The runtime must know the difference.
11. Verification Is a Special Dependency
Many agent systems treat verification as just another optional service.
That is dangerous.
Suppose the verifier is overloaded.
The naive platform keeps producing more speculative solutions while the verification queue grows.
Eventually:
solutions produced >> solutions that can be verified
The platform now accumulates untrusted work.
Instead:
verification capacity low
↓
reduce speculative generation
↓
reduce search width
↓
reduce critic fan-out
↓
preserve verifier capacity
This follows the protected-verification principle from Steps 16 and 21.
If proof capacity falls, solution-generation capacity should usually fall with it.
Otherwise the platform optimizes production of claims rather than verified outcomes.
12. Verification Failure Must Not Become Success
If the verifier is unavailable, distinguish:
FAIL
from:
UNKNOWN
A verifier timeout does not prove the candidate is wrong.
It also does not prove the candidate is right.
For example:
candidate produced
↓
verification unavailable
↓
UNKNOWN
The platform may:
- queue verification for later,
- return an explicitly unverified result,
- keep the task pending,
- or stop safely.
What it must not do is convert missing evidence into PASS.
13. Model Failure Has Multiple Forms
A model dependency can fail through:
- transport errors,
- rate limiting,
- latency spikes,
- malformed structured output,
- context-window rejection,
- policy refusal,
- empty responses,
- quality regression,
- or incorrect tool usage.
Only some of these are infrastructure failures.
For example:
HTTP 503
may justify a provider-level breaker.
But:
model produced invalid JSON
may justify a structured-output recovery path or model-specific quality breaker.
Similarly:
model returned syntactically valid answer
but verifier false-pass rate rose sharply
is a correctness health problem.
Dependency health should include semantic quality where measurable, not only HTTP status.
14. Browser Failure Needs Domain-Aware Isolation
Browser agents are especially vulnerable to broad failure classification.
Suppose:
site A blocks automation
site B works normally
A global browser breaker would unnecessarily disable all browser work.
Instead the failure domain may be:
browser_pool × target_domain × operation
Examples:
navigate(example.com)
submit_form(example.com)
download(example.com)
Different actions may have different failure modes.
A read-only navigation failure should not necessarily disable a separate internal-browser workflow.
Again:
Break at the narrowest boundary that correctly contains the failure.
15. Research Agents Need Source-Level Health
A research agent may use:
search API
web fetch
PDF parser
vector store
primary-source database
LLM synthesis
If the search API degrades, the system can potentially fall back to:
- cached search results,
- known primary sources,
- alternate search providers,
- direct source navigation,
- or a narrower answer with explicit evidence limitations.
But if the primary-source database is unavailable for a claim that specifically requires it, the correct outcome may be:
INSUFFICIENT_EVIDENCE
Graceful degradation should preserve evidence semantics.
16. Coding Agents Need Workspace Bulkheads
Coding-agent failures can propagate through shared workspaces.
Bad design:
multiple speculative branches
↓
shared working tree
↓
partial edits / lock contention / corruption
Better:
branch A → worktree A
branch B → worktree B
branch C → worktree C
Then:
select verified candidate
↓
revalidate base state
↓
apply or merge
This extends the speculative isolation from Step 19 and the fencing discipline from Step 20.
If one sandbox cluster is unhealthy, the circuit breaker should redirect or defer sandbox-dependent work without preventing read-only repository analysis.
17. DevOps Agents Need the Strictest Degradation Policy
A DevOps agent may control:
- deployments,
- restarts,
- traffic shifts,
- scaling,
- rollbacks,
- database migrations,
- and infrastructure changes.
When dependencies are degraded, the system should generally become more conservative, not more autonomous.
For example:
observability incomplete
↓
block risky mutation
not:
observability incomplete
↓
let model guess
A useful DevOps service mode might be:
NORMAL
diagnostics + controlled mutations
DEGRADED_OBSERVABILITY
diagnostics only
no irreversible mutations
DEGRADED_VERIFIER
mutation blocked
EMERGENCY
deterministic runbooks only
Graceful degradation here means reducing authority.
18. The Platform Needs a Dependency Registry
A production runtime should know what it depends on.
from dataclasses import dataclass
@dataclass(frozen=True)
class DependencySpec:
name: str
required: bool
fallback: str | None
bulkhead: str
timeout_s: float
retry_limit: int
breaker_key: str
For example:
DEPENDENCIES = {
"frontier_model": DependencySpec(
name="frontier_model",
required=False,
fallback="local_model",
bulkhead="model_frontier",
timeout_s=30,
retry_limit=1,
breaker_key="provider:model",
),
"verifier": DependencySpec(
name="verifier",
required=True,
fallback=None,
bulkhead="verification",
timeout_s=20,
retry_limit=2,
breaker_key="verifier_cluster",
),
}
This makes failure behavior explicit instead of scattering it across exception handlers.
19. Health Decisions Need Hysteresis
Without hysteresis, systems oscillate.
Imagine:
latency high
→ breaker opens
→ load drops
→ latency improves
→ breaker closes
→ load surges
→ latency high
→ breaker opens
This is flapping.
Use asymmetric transitions.
For example:
open after:
5 failures in 20 seconds
close after:
10 successful probes
and p95 latency below threshold
for 60 seconds
Recovery should require stronger evidence than initial degradation detection.
This is the same reason Step 21 used hysteresis for platform overload modes.
20. Probe Traffic Must Be Bounded
When an OPEN breaker transitions toward recovery, do not release all waiting work at once.
Use bounded probes.
OPEN
↓ cooldown
HALF_OPEN
↓
allow 1 probe
↓
success
↓
allow 3 probes
↓
stable
↓
CLOSED
This avoids a thundering herd.
Probe traffic is effectively a small experiment:
has the dependency recovered enough to accept normal load?
Treat it as such.
21. Queueing Behind an Open Breaker Is Often Wrong
Suppose the frontier model breaker is OPEN.
A naive design queues every new request waiting for recovery.
Now the queue grows indefinitely.
When the dependency returns, thousands of requests arrive simultaneously.
Instead admission should choose explicitly:
fallback
queue bounded
shed
defer
reject
This connects directly to Step 21.
A circuit breaker should inform admission control.
It should not simply redirect failures into an unbounded backlog.
22. Failure Containment Needs a Failure Taxonomy
Useful production labels include:
DEPENDENCY_TIMEOUT
DEPENDENCY_RATE_LIMITED
DEPENDENCY_UNAVAILABLE
DEPENDENCY_SEMANTIC_FAILURE
CIRCUIT_OPEN
BULKHEAD_EXHAUSTED
RETRY_BUDGET_EXHAUSTED
FALLBACK_UNAVAILABLE
FALLBACK_INSUFFICIENT
STALE_DATA_UNSAFE
VERIFIER_UNAVAILABLE
VERIFICATION_UNKNOWN
RECOVERY_PROBE_FAILED
DEPENDENCY_FLAPPING
CASCADE_PREVENTED
These labels are far more useful than:
agent_error
because they tell you which containment mechanism failed or succeeded.
23. Measure Containment, Not Just Availability
Important metrics include:
- breaker-open rate,
- breaker false-open rate,
- breaker false-close rate,
- half-open recovery success,
- fallback success rate,
- fallback verified-success delta,
- bulkhead saturation,
- retry amplification factor,
- retry success rate,
- stale-read rescue rate,
- stale-read regression rate,
- verification-starvation rate,
- cascade-prevention count,
- dependency-attributed latency,
- degraded-mode verified success,
- and recovery time.
You also want:
cost while dependency unhealthy
because a platform that “survives” by spending 20× more is not necessarily healthy.
24. Verify the Fallback Too
Suppose your primary model fails and the fallback model answers successfully.
That only proves:
fallback produced output
It does not prove:
fallback met task requirements
So fallback outcomes should go through the same external verifier whenever possible.
primary unavailable
↓
fallback
↓
external verification
↓
PASS / FAIL / UNKNOWN
This also gives you real evidence about whether the fallback is worth keeping.
25. Graceful Degradation Should Be Tested Deliberately
Do not wait for production failure to learn whether the architecture degrades safely.
Inject failures.
Examples:
frontier model returns 503 for 5 minutes
browser latency ×10
retrieval returns stale data
verifier unavailable
DB pool saturated
search API rate limited
sandbox cluster unavailable
half-open probes fail intermittently
Then measure:
verified success
false success
UNKNOWN rate
latency
cost
queue growth
retry amplification
fallback usage
verification starvation
recovery time
The key question is not:
did the system stay up?
It is:
Did the system preserve the right guarantees while losing optional capability?
26. A Failure-Containment Runtime
A simplified dependency wrapper might look like this:
from dataclasses import dataclass
from typing import Callable, TypeVar
T = TypeVar("T")
class DependencyUnavailable(RuntimeError):
pass
@dataclass
class DependencyRuntime:
breaker: CircuitBreaker
max_in_flight: int
in_flight: int = 0
def call(self, fn: Callable[[], T]) -> T:
if not self.breaker.allow():
raise DependencyUnavailable("circuit open")
if self.in_flight >= self.max_in_flight:
raise DependencyUnavailable("bulkhead exhausted")
self.in_flight += 1
try:
result = fn()
self.breaker.record_success()
return result
except Exception:
self.breaker.record_failure()
raise
finally:
self.in_flight -= 1
Production code would add:
- concurrency-safe counters,
- rolling windows,
- error classification,
- structured trace events,
- retry policy,
- timeout handling,
- fallback routing,
- health telemetry,
- and policy versioning.
But the important architecture remains small and understandable.
27. Connect Failure State to the Global Scheduler
Dependency health should be an input to platform scheduling.
Suppose browser capacity drops by 80%.
The scheduler should not continue admitting browser-heavy workloads at the same rate.
browser health degraded
↓
effective capacity reduced
↓
admission changes
↓
queue bounded
↓
optional browser work shed/deferred
Likewise, if the verifier cluster is degraded:
verification capacity reduced
↓
reduce solution generation
↓
reduce search fan-out
↓
protect remaining verifier capacity
This closes the loop between Steps 21 and 22.
28. Failure State Must Also Reach the Per-Run Scheduler
The platform may still admit a task while some dependencies are degraded.
The run-level scheduler should adapt.
Example:
frontier model breaker open
↓
route to local model where capability allows
↓
reduce search width
↓
reserve verification
Or:
browser unavailable
↓
use cached read-only evidence if valid
↓
otherwise return UNKNOWN
This is not autonomous improvisation.
It is explicit policy.
29. Keep Safety Outside the Fallback Layer
Failure is exactly when shortcuts become tempting.
Do not let degraded mode disable:
- authorization,
- sandbox boundaries,
- fencing,
- tenant isolation,
- prohibited tool rules,
- mandatory acceptance criteria,
- or audit logging.
If the platform cannot operate while preserving those controls, the correct action is:
stop
not:
relax safety because the dependency is down
30. The Simplest Reliable Architecture Wins
Circuit breakers, bulkheads, fallbacks and degradation modes add complexity.
Do not add them everywhere by default.
Start with:
timeout
bounded retry
bounded concurrency
explicit error
Then measure failures.
Add a breaker when repeated calls to a known-unhealthy dependency create real cost or latency.
Add a bulkhead when one dependency can consume capacity needed by unrelated work.
Add fallback paths when they preserve enough capability to matter.
Add degraded service modes when the platform has meaningful optional work to shed.
The same rule from the rest of this series still applies:
Every reliability mechanism should target a measured failure and earn the operational complexity it introduces.
31. A Practical Failure-Containment Checklist
Before calling an agent platform resilient, ask:
- Can one dependency consume all worker or verifier capacity?
- Are retries bounded, classified and jittered?
- Can an unhealthy dependency be temporarily removed from routing?
- Are breaker scopes narrow enough to avoid unnecessary outages?
- Are half-open probes bounded?
- Can degraded dependencies change admission control?
- Are fallback capability requirements explicit?
- Can stale data be distinguished from safe stale data?
- Is verification capacity protected?
- Does missing verification produce UNKNOWN rather than PASS?
- Can speculative work be reduced before critical work?
- Are external provider limits modeled as resources?
- Is dependency health visible in trajectory traces?
- Are degraded-mode decisions versioned and replayable?
- Can failure modes be injected deliberately?
- Are safety and authorization preserved during degradation?
If several answers are no, the architecture is probably relying on the happy path.
32. The Architecture So Far
The advanced-agent runtime has now moved a long way beyond “call an LLM in a loop.”
reasoning mechanisms
↓
search / debate / routing
↓
external verification
↓
benchmarking
↓
trajectory observability
↓
trajectory learning
↓
policy optimization
↓
dynamic budget scheduling
↓
typed uncertainty
↓
value of information
↓
speculative parallel execution
↓
distributed coordination
↓
platform admission / fairness / backpressure
↓
failure containment / graceful degradation
The common thread is not autonomy.
It is controlled computation under uncertainty with explicit evidence and failure boundaries.
What Comes Next
Circuit breakers and bulkheads help the platform survive unhealthy dependencies.
But another reliability problem remains.
What happens when the system is technically healthy, but its outputs are drifting?
A new model version changes behavior.
A router slowly starts preferring the wrong expert.
A verifier becomes too permissive.
A retrieval corpus changes.
A previously successful policy degrades as the task distribution shifts.
Nothing crashes.
Nothing returns HTTP 500.
The platform is still getting worse.
The next step is therefore:
behavioral drift detection and automatic rollback — monitoring verified outcomes, route distributions, verifier calibration and policy behavior across versions so silent quality regressions are caught before they become the new normal.