# Advanced Agents From First Principles 05: Is One Model Doing Everything? Build a Mixture of Experts at the Agent Level
# Book: Advanced Agents From First Principles
# Chapter URL: https://programmer.ie/books/advanced-agents-from-first-principles/05-chapter/
# Prose fingerprint (sha256, normalized): e3deef21ca8043e6d44f68de91fb4f4fc07a4a73694e0f173086ad39860f4356
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

A common agent architecture starts simply:
request ↓ model ↓ action That simplicity is valuable.
It should be your default.
But eventually you may notice something strange.
The same model is being asked to do everything:
classify the task, search documentation, reason about code, write SQL, review a patch, summarize logs, judge another model, decide whether a deployment is safe, and answer simple questions that did not require an expensive model in the first place. At that point the problem may no longer be:
How do I make the model smarter?
It may be:
Why am I sending every problem to the same expert?
That is where an agent-level Mixture of Experts becomes useful.
But the phrase is easy to misunderstand.
This post is not primarily about the Mixture-of-Experts layers used inside large neural networks.
It is about something much simpler and much more directly useful when building agent systems:
task ↓ router ├──→ local model ├──→ frontier model ├──→ code specialist ├──→ retrieval specialist ├──→ deterministic tool ├──→ critic └──→ verifier The important idea is not that more experts are automatically better.
The important idea is that different work should be routed to the mechanism most suited to doing it.
And, as with every technique in this series, that additional complexity has to earn its place.
The Search Problem This Post Solves People building production agents eventually search for questions like:
How do I route tasks between multiple LLMs? How do I use a local model and GPT/Claude together? How do I choose the best model for each prompt? Why is my agent sending easy tasks to an expensive model? Why does my multi-agent router choose the wrong specialist? How do I build a mixture of agents? How do I route between tools and models? How do I use specialist AI agents? Should I use multiple AI models in one application? How do I reduce LLM costs without reducing quality? Why does my expert router collapse onto one model? How do I know whether agent specialization actually helps? Those look like different questions.
Architecturally, they are usually the same problem:
many possible capabilities ↓ one incoming task ↓ which capability should receive it? That is a routing problem.
Neural MoE and Agent MoE Are Different Things Before building anything, we need to separate two ideas that share a name.
Neural Mixture of Experts Inside a neural network, a router may decide which learned expert layers process a token or representation:
input representation ↓ router / | \ ↓ ↓ ↓ E1 E2 E3 \ | / ↓ network output The experts are parts of one learned model.
The routing happens inside the neural architecture.
Agent-Level Mixture of Experts At the software-system level, our experts can be entirely different components:
user task ↓ router ├── local LLM ├── frontier LLM ├── code model ├── retrieval system ├── SQL specialist ├── deterministic calculator ├── browser agent └── verifier These experts may differ in:
model family, model size, prompt, tools, memory, cost, latency, context window, domain specialization, or even whether they are an LLM at all. This post is about the second architecture.
The Simplest Possible Router We do not need machine learning to begin.
Suppose our application receives four kinds of work:
calculation coding research ordinary conversation The first router could literally be ordinary Python:
from dataclasses import dataclass from typing import Literal ExpertName = Literal[ "calculator", "code_model", "research_model", "general_model", ] @dataclass class RouteDecision: expert: ExpertName reason: str def route(task: str) -> RouteDecision: lower = task.lower() if "calculate" in lower or "sum" in lower: return RouteDecision( expert="calculator", reason="Task appears deterministic and arithmetic.", ) if "python" in lower or "bug" in lower or "code" in lower: return RouteDecision( expert="code_model", reason="Task appears to require code reasoning.", ) if "research" in lower or "sources" in lower: return RouteDecision( expert="research_model", reason="Task appears to require evidence gathering.", ) return RouteDecision( expert="general_model", reason="No specialist route matched.", ) That is already a mixture-of-experts system.
Nothing about MoE requires a neural router.
This matters because the cheapest router that works is usually the best starting point.
Why Routing Can Beat One Giant Generalist Imagine every task is currently sent to your strongest model.
The architecture is:
all tasks ↓ expensive frontier model That can be perfectly reasonable.
But suppose your workload looks like this:
55% straightforward classification 20% retrieval 15% code reasoning 8% difficult reasoning 2% genuinely hard edge cases Using the most expensive model for 100% of requests may be wasteful.
A router could instead create:
request ↓ router ├─ 55% → cheap classifier/local model ├─ 20% → retrieval path ├─ 15% → code specialist ├─ 8% → medium model └─ 2% → frontier model The potential gain is not only financial.
Specialists can also have:
narrower prompts, fewer irrelevant tools, tighter schemas, smaller contexts, stronger deterministic checks, and more appropriate verification. So routing may improve both efficiency and reliability.
But only if the router is good enough.
The Router Is Now Part of Your Failure Surface Once we add routing, we have created a new subsystem that can fail.
Before routing:
model failure After routing:
routing failure or expert failure or verification failure This distinction is critical.
Suppose a coding task gets sent to the research specialist and fails.
The code expert may be excellent.
The system still fails because the router never reached it.
So we need to measure:
routing quality separately from expert quality Define Experts by Capability, Not Personality One common mistake in multi-agent systems is defining roles like:
The Architect The Skeptic The Genius The Researcher The Visionary Those names sound interesting.
They tell the runtime almost nothing useful.
Prefer capability contracts.
For example:
from dataclasses import dataclass, field @dataclass class ExpertSpec: name: str capabilities: set[str] cost_tier: int latency_tier: int tools: set[str] = field(default_factory=set) notes: str = "" Then:
experts = [ ExpertSpec( name="local_classifier", capabilities={"classify", "route", "extract"}, cost_tier=1, latency_tier=1, ), ExpertSpec( name="code_specialist", capabilities={"code", "debug", "review"}, cost_tier=2, latency_tier=2, tools={"read_file", "search_code", "run_tests"}, ), ExpertSpec( name="research_specialist", capabilities={"research", "retrieve", "synthesize"}, cost_tier=2, latency_tier=2, tools={"search", "open_source", "extract_evidence"}, ), ExpertSpec( name="frontier_reasoner", capabilities={"reason", "ambiguous", "hard_case"}, cost_tier=4, latency_tier=4, ), ] Now the router has something concrete to reason over.
Routing Is Classification Over Capabilities At its simplest, routing is classification:
input task ↓ features ↓ classify required capability ↓ select compatible expert For example:
"Fix the failing pytest test" ↓ capability = code/debug/test ↓ code_specialist Or:
"What was our revenue last quarter?" ↓ capability = structured_data_query ↓ SQL / analytics expert Or:
"What is 17.4 × 8.2?" ↓ capability = deterministic_math ↓ calculator Notice the final example.
The best expert is not necessarily an LLM.
That is one of the most useful ideas in agent-level MoE.
Deterministic Experts Belong in the Mixture Too A mixture of experts can contain:
LLMs retrievers calculators SQL engines static analyzers compilers test runners rule engines search systems specialized ML models If a deterministic system can answer a question more reliably, it should often win the route.
For example:
def preferred_expert(capability: str) -> str: deterministic = { "arithmetic": "calculator", "sql": "database", "syntax_check": "parser", "tests": "test_runner", } if capability in deterministic: return deterministic[capability] return "llm" This is not less agentic.
It is better systems engineering.
A Router Should Return More Than a Name A production route decision should be inspectable.
For example:
from dataclasses import dataclass @dataclass class RouteDecision: expert: str confidence: float required_capabilities: list[str] alternatives: list[str] reason: str Then the runtime can distinguish:
high-confidence route from ambiguous route That enables adaptive behavior.
Low Confidence Should Change Control Flow Suppose the router returns:
RouteDecision( expert="code_specialist", confidence=0.96, required_capabilities=["code", "debug"], alternatives=[], reason="Explicit request to debug Python code.", ) That is probably safe to route directly.
But:
RouteDecision( expert="research_specialist", confidence=0.52, required_capabilities=["research", "code"], alternatives=["code_specialist"], reason="Task requires both repository inspection and external evidence.", ) That uncertainty should matter.
Possible responses include:
route to multiple experts or ask a stronger router or use a generalist or split the task or route sequentially Uncertainty is not merely metadata.
It should affect the architecture.
Hard Routing The simplest MoE architecture chooses exactly one expert:
request ↓ router ↓ expert B ↓ result This is hard routing.
It is cheap and easy to understand.
Its main failure mode is obvious:
wrong route = potentially lost task If the router chooses the wrong expert, the correct expert never gets a chance.
Top-K Routing Instead of selecting one expert, we can select several:
request ↓ router ├→ expert B └→ expert D ↓ aggregate / verify This is useful when:
route confidence is low, two capabilities are genuinely required, specialists provide complementary evidence, or the cost of a wrong route is high. But it also increases:
calls, latency, aggregation complexity, and correlated failure risk. Do not make k=3 your default merely because it feels safer.
Measure it.
Sequential Routing Some tasks are not best solved by parallel experts.
They require a sequence.
For example:
research specialist ↓ extract evidence ↓ code specialist ↓ implement change ↓ verifier This is closer to a routed workflow than classical MoE.
But the same expert-selection principle applies.
The important question is:
Which capability should receive the current state next?
That means routing can happen repeatedly during one trajectory.
Routing Is Not the Same as Multi-Agent Conversation A mixture of experts does not require experts to talk to each other.
This is perfectly valid:
router ↓ expert ↓ verifier No committee.
No debate.
No agent group chat.
That simplicity is often an advantage.
We should only introduce agent-to-agent communication when information genuinely has to flow between specialists.
The Local Model + Frontier Model Pattern One of the most useful real-world MoE patterns is:
cheap/local model first ↓ can it solve task confidently? ├─ yes → verify → return └─ no → frontier model This architecture can dramatically reduce expensive calls if many tasks are easy.
A simple escalation policy might be:
def choose_model( complexity: float, uncertainty: float, previous_failure: bool, ) -> str: if previous_failure: return "frontier" if complexity > 0.75: return "frontier" if uncertainty > 0.60: return "frontier" return "local" But remember:
model confidence ≠ correctness The escalation should ideally use external verification too.
Verification-Driven Escalation A stronger design is:
local expert ↓ verification ├─ PASS → return ├─ FAIL → escalate └─ UNKNOWN → escalate Now the system does not need to trust the local model’s self-assessment.
For example:
def route_after_local(result, verification): if verification.status == "PASS": return "accept" if verification.status in {"FAIL", "UNKNOWN"}: return "frontier_model" This connects directly to the verification architecture from the core agents series.
Cheap Experts First Is a Compute Policy We can think of MoE routing as compute allocation.
Suppose:
local model = 1 unit specialist model = 3 units frontier model = 12 units Then the router is effectively deciding:
How much inference compute does this task deserve? That connects agent-level MoE to the broader theme of the advanced series:
Advanced agent architecture is often really compute allocation under uncertainty.
Cost-Aware Routing We can explicitly include cost in route selection.
For example:
@dataclass class CandidateRoute: expert: str predicted_success: float estimated_cost: float estimated_latency_ms: float def utility(route: CandidateRoute) -> float: return ( route.predicted_success - 0.03 * route.estimated_cost - 0.0001 * route.estimated_latency_ms ) Then:
chosen = max(routes, key=utility) This is intentionally simplistic.
The important idea is that route selection can optimize more than raw quality.
Production systems care about:
success, cost, latency, privacy, locality, tool access, context limits, and safety constraints. Capability Constraints Should Come Before Preference Scores Do not let a cheap expert win a route it cannot perform.
First filter by hard constraints:
def compatible(expert: ExpertSpec, required: set[str]) -> bool: return required.issubset(expert.capabilities) Then rank compatible experts.
Architecture:
all experts ↓ hard capability filter ↓ compatible experts ↓ soft scoring ↓ selected route Hard constraints should not be traded away by a fuzzy score.
Experts Need Sharp Boundaries Suppose we define these experts:
code_expert software_expert programming_expert engineering_expert technical_expert What should the router do?
These experts overlap heavily.
The router now has an unnecessarily difficult classification problem.
Prefer boundaries like:
repository_retrieval patch_generation test_diagnosis architecture_review security_review Or, at model level:
cheap_generalist code_specialist long_context_researcher frontier_reasoner The experts should have a reason to exist.
Expert Collapse A common MoE failure is expert collapse.
The router discovers one expert that performs reasonably well and sends nearly everything there.
You intended:
25% A 25% B 25% C 25% D You get:
2% A 3% B 92% C 3% D This is not automatically wrong.
Perhaps C really is best.
But it is worth investigating.
Measure expert utilization:
from collections import Counter counts = Counter(route.expert for route in route_history) Then inspect:
task distribution, expert quality, router calibration, expert overlap, and whether some experts are unnecessary. Do not force equal traffic simply to make the architecture look balanced.
Redundant Experts Are Another Failure Mode Suppose two experts consistently produce nearly identical outputs.
Then the second expert may not be providing meaningful specialization.
Measure marginal contribution.
For expert E:
system success with E - system success without E If the difference is near zero, ask why E exists.
This is a powerful ablation.
Routing Accuracy Is Not Enough Imagine your router selects the “correct” expert 95% of the time according to labels.
That sounds excellent.
But what if the expert itself performs poorly?
We need several metrics.
routing accuracy expert success conditional on route overall verified success cost latency The real objective is not:
correct expert label It is:
verified task success Routing accuracy is only an intermediate metric.
Build a Routing Confusion Matrix Suppose the expected classes are:
code research data general Then track:
expected → selected Example:
selected expected code research data general ---------------------------------------- code 91 4 2 3 research 7 84 4 5 data 5 3 89 3 general 6 4 2 88 Immediately we can see:
research → code is a recurring confusion.
That gives us something concrete to fix.
Why the Router Makes Mistakes Common causes include:
1. Overlapping expert descriptions code expert: handles technical programming tasks research expert: handles technical investigation Many tasks match both.
2. Missing task context The router receives:
"Fix this" instead of repository state, failure type, and user objective.
3. Too many experts Every added route increases the classification space.
4. Bad labels Your benchmark may disagree with what actually produces the best verified outcome.
5. Stale expert capabilities The router thinks an expert supports a tool or context size it no longer supports.
6. Cost dominates too strongly The optimizer repeatedly selects a cheap but inadequate expert.
Route by State, Not Just by User Prompt Agent routing becomes much more powerful when the router sees runtime state.
For example:
@dataclass class AgentState: task: str step: int failures: list[str] evidence: list[str] last_expert: str | None verification_status: str | None Then routing can change during the trajectory.
Example:
initial task ↓ local code model ↓ tests still fail ↓ debug specialist ↓ verification unknown ↓ frontier reasoner This is much more interesting than one-time prompt classification.
Failure-Aware Routing Repeated failure is useful routing evidence.
Suppose the same expert has failed twice.
Do not simply send the task back unchanged.
def should_escalate(state: AgentState) -> bool: return len(state.failures) >= 2 Then:
same expert fails repeatedly ↓ change expert / model / strategy This connects routing to the loop-control ideas from the earlier series.
A Full Minimal Agent-Level MoE Runtime Here is a small provider-independent skeleton.
from dataclasses import dataclass, field from typing import Callable, Any @dataclass class Expert: name: str capabilities: set[str] run: Callable[[dict], Any] cost_tier: int = 1 @dataclass class RouteDecision: expert: str confidence: float alternatives: list[str] = field(default_factory=list) reason: str = "" @dataclass class ExpertResult: expert: str output: Any verified: bool | None = None class ExpertRegistry: def __init__(self): self._experts: dict[str, Expert] = {} def register(self, expert: Expert) -> None: self._experts[expert.name] = expert def get(self, name: str) -> Expert: return self._experts[name] def compatible(self, capabilities: set[str]) -> list[Expert]: return [ expert for expert in self._experts.values() if capabilities.issubset(expert.capabilities) ] class AgentMoE: def __init__(self, registry, router, verifier): self.registry = registry self.router = router self.verifier = verifier def run(self, task: dict) -> ExpertResult: required = set(task.get("required_capabilities", [])) candidates = self.registry.compatible(required) if not candidates: raise RuntimeError( f"No expert supports capabilities: {sorted(required)}" ) decision = self.router(task, candidates) expert = self.registry.get(decision.expert) output = expert.run(task) verification = self.verifier(task, output) if verification == "PASS": return ExpertResult( expert=expert.name, output=output, verified=True, ) if verification in {"FAIL", "UNKNOWN"}: for alternative_name in decision.alternatives: alternative = self.registry.get(alternative_name) output = alternative.run(task) verification = self.verifier(task, output) if verification == "PASS": return ExpertResult( expert=alternative.name, output=output, verified=True, ) return ExpertResult( expert=expert.name, output=output, verified=False, ) This gives us the essential architecture:
capability filter ↓ router ↓ expert ↓ verifier ↓ fallback / escalation Routing Can Be Deterministic Do not immediately train a router.
A rule-based router can be excellent when the domains are clear.
Example:
def deterministic_router(task, candidates): capabilities = set(task["required_capabilities"]) if "arithmetic" in capabilities: target = "calculator" elif "code" in capabilities: target = "code_specialist" elif "research" in capabilities: target = "research_specialist" else: target = "general_model" alternatives = [ expert.name for expert in candidates if expert.name != target ] return RouteDecision( expert=target, confidence=1.0, alternatives=alternatives, reason="Deterministic capability rule.", ) If this solves 95% of your routing problem, keep it.
LLM Router When task boundaries are fuzzy, an LLM can classify the request.
But the router itself should have a constrained output schema.
For example:
{ "required_capabilities": ["code", "debug"], "selected_expert": "code_specialist", "confidence": 0.83, "alternatives": ["frontier_reasoner"] } Then validate that:
the expert exists, the expert supports the capabilities, confidence is within bounds, alternatives exist, the route is authorized. The router should not be allowed to invent experts.
Learned Router Eventually you may have enough routing evidence to train a small classifier.
Inputs might include:
task embedding repository type tool requirements context length failure history cost budget latency budget Output:
P(expert | task, state) This can be valuable at high volume.
But it creates a new model that must itself be evaluated, versioned, calibrated, and monitored.
Do not build it because “MoE should have a learned router.”
Build it because the routing dataset shows deterministic and prompt-based routing have become the bottleneck.
Router Training Data Should Come From Outcomes A subtle point:
The best training label is not necessarily:
which expert sounds appropriate? It is closer to:
which expert produced the best verified outcome for this task under the relevant cost/latency constraints? That means routing data should ideally contain:
@dataclass class RoutingEvidence: task_id: str expert: str verified_success: bool cost: float latency_ms: float failure_type: str | None Now router improvement becomes evidence-driven.
Expert Selection Can Be Contextual Bandit-Like Once the system gathers outcome evidence, routing begins to resemble a contextual bandit problem:
context ↓ choose expert ↓ observe reward/outcome ↓ update routing policy That does not mean you should immediately deploy online learning.
But it gives us a useful conceptual model:
routing is a decision under uncertainty with observable downstream consequences This becomes important later when we discuss adaptive agents.
Specialist Judges Are Experts Too Suppose we generate code with one model.
We might route the result to specialists:
patch ↓ syntax verifier ↓ test runner ↓ security reviewer ↓ performance reviewer These are not necessarily candidate generators.
They are expert evaluators.
So an agent MoE can exist on both sides:
generation experts and evaluation experts This is especially useful when failure classes differ significantly.
But Do Not Build a Reviewer Army Without Evidence Imagine:
security critic style critic architecture critic correctness critic performance critic maintainability critic simplicity critic That looks thorough.
It may simply multiply cost.
Ask:
What additional verified defects does each reviewer find? Measure marginal defect discovery.
If two reviewers find the same issues almost every time, one may be redundant.
Expert Diversity Should Be Functional Useful diversity comes from different capabilities or evidence channels.
For example:
LLM reviewer static analyzer test runner security scanner runtime benchmark Those are genuinely different experts.
Less useful:
same model + "be a reviewer" same model + "be a critic" same model + "be skeptical" Prompt-role diversity can help.
It should not be confused with independent evidence.
Agent MoE for Coding Software Coding agents are a natural application.
A useful mixture might look like:
engineering task ↓ router ├→ repository retrieval ├→ code generator ├→ test diagnosis ├→ architecture reviewer ├→ security reviewer └→ frontier escalation Example route:
"Why is CI failing?" ↓ CI/log specialist ↓ identify failing test ↓ code specialist ↓ patch ↓ test verifier The application is not “several agents talking.”
It is specialized engineering capabilities connected by explicit routing.
Agent MoE for Research Software A research system may have:
query classifier web retrieval specialist paper retrieval specialist source-quality evaluator claim extractor synthesis model citation verifier A route might be:
question ↓ requires current evidence? ├─ no → internal/general model └─ yes → retrieval specialist ↓ evidence ↓ synthesis ↓ citation verifier Again, specialization is driven by the work.
Agent MoE for Customer Support A support architecture might route by case type:
incoming case ↓ classifier ├→ billing ├→ shipping ├→ technical support ├→ account access └→ human escalation Each route can have different:
tools, permissions, policies, verification, and risk limits. This is a strong reason to route: not every specialist should have access to every action.
Agent MoE for Data and Analytics A data agent might use:
question ↓ router ├→ SQL generator ├→ dataframe analyst ├→ statistical model ├→ chart generator └→ narrative summarizer Important distinction:
"calculate revenue by quarter" should probably route toward deterministic data execution.
Whereas:
"explain why revenue changed" may require synthesis after the computation.
One request can therefore cross multiple experts sequentially.
Agent MoE for DevOps and Incident Response An incident system could contain:
log specialist metrics specialist configuration specialist network specialist deployment specialist rollback verifier A route might depend on evidence:
latency spike ↓ metrics specialist ↓ DB saturation detected ↓ database specialist ↓ remediation candidate ↓ verification This is state-dependent expert routing.
Agent MoE for Browser Automation A browser system may divide work between:
page-state parser navigation planner form specialist extraction specialist visual fallback model Most pages may be handled cheaply from DOM structure.
Only difficult visual states may need the expensive vision model.
That gives us:
DOM-first ↓ verification ↓ vision escalation only when necessary A classic adaptive-compute MoE pattern.
Application Matrix Software Possible experts Useful routing signal Strong verification Coding agent retrieval, generator, debugger, reviewer task/failure type tests, build, static analysis Research agent search, source evaluator, synthesis evidence need/source type source support/citations Support agent billing, technical, account, escalation case classification backend state/policy checks Data agent SQL, stats, dataframe, narrative data operation required query results/invariants DevOps agent logs, metrics, DB, deployment observed failure signal health checks/metrics Browser agent DOM, navigation, forms, vision page state resulting DOM/page state The important pattern is:
route based on task/state verify using the environment The Router Should Not Be the Final Judge Suppose the router says:
expert = code_specialist confidence = 0.98 That does not prove the code specialist will succeed.
The architecture still needs:
route ↓ execute ↓ verify Routing confidence is about routing.
It is not task-success confidence.
Expert Confidence Is Also Not Enough The expert may say:
"I am 95% confident the issue is fixed." The test suite may say:
FAIL The test suite wins.
The evidence hierarchy remains:
environment evidence > model confidence Fallback Should Be Explicit Production routing needs a fallback policy.
For example:
route expert A ↓ verification FAIL ↓ expert B ↓ verification UNKNOWN ↓ frontier generalist ↓ verification Avoid unlimited fallback chains.
Set budgets:
MAX_EXPERT_ATTEMPTS = 3 MAX_TOTAL_COST = 2.50 MAX_LATENCY_SECONDS = 30 The system must eventually stop.
Track Why Escalation Happened Do not log only:
model = frontier Log:
initial_expert = local_code reason_for_escalation = verification_failed failed_check = test_user_creation frontier_expert = frontier_code This makes expensive routes explainable.
It also lets you improve the cheap path later.
Routing Telemetry At minimum, record:
@dataclass class RouteTrace: task_id: str selected_expert: str confidence: float alternatives: list[str] route_reason: str verified_success: bool | None cost: float latency_ms: float escalated: bool escalation_reason: str | None Now you can ask:
Which expert gets the most traffic? Which expert has the best verified-success rate? Which routes fail most often? Where does escalation happen? Which expert is expensive but rarely necessary? Which specialist contributes unique wins? Metrics That Actually Matter Useful metrics include:
Router metrics routing accuracy routing calibration route entropy expert utilization Expert metrics verified success by expert failure type by expert latency by expert cost by expert System metrics overall verified success cost per verified success p50/p95 latency escalation rate fallback success rate Specialization metrics marginal success contribution unique defect discovery expert redundancy These tell us whether MoE is actually doing useful work.
Route Entropy One simple diagnostic is route entropy.
If every request goes to the same expert, route entropy is low.
That may mean:
expert collapse, the router is biased, or one expert genuinely dominates. If every request is distributed almost uniformly, route entropy is high.
That may mean:
tasks are genuinely diverse, or the router is uncertain/noisy. The metric is diagnostic, not an objective.
Do not optimize entropy for its own sake.
When More Experts Make the System Worse Adding experts can reduce reliability because it increases:
routing ambiguity configuration complexity observability burden fallback complexity benchmarking surface Suppose success is:
one model = 89% 3 experts = 92% 8 experts = 91% 16 experts = 88% The sophisticated architecture lost.
More specialists did not create more specialization.
They created a harder routing problem.
Do Not Add an Expert Without a Failure It Owns A useful rule:
Every expert should own a recognizable failure class or capability boundary.
For example:
Expert: test_diagnosis Failure it addresses: General model changes code before understanding the failing test. Or:
Expert: long_context_retrieval Failure it addresses: General model loses relevant repository context in large codebases. Or:
Expert: frontier_escalation Failure it addresses: Local model fails on the small subset of genuinely difficult tasks. If you cannot write that sentence, you may not need the expert.
Ablate Experts One at a Time A simple experiment:
full system full system - expert A full system - expert B full system - expert C Measure:
verified success cost latency failure coverage If removing an expert changes nothing, that is strong evidence it may be unnecessary.
Compare Against the One-Model Baseline The most important experiment is still:
one strong model vs routed mixture A useful table:
Architecture Verified success Avg cost p95 latency One strong model 91% $0.18 4.2s Local only 78% $0.02 1.1s Local → frontier escalation 92% $0.07 3.0s 5-expert router 93% $0.10 4.8s These numbers are illustrative.
The question is not:
Which architecture sounds advanced? It is:
Which architecture produces the best verified outcome under our actual constraints? Run an Oracle Routing Experiment One particularly useful experiment is oracle routing.
For every benchmark task, run all candidate experts offline.
Then determine:
which expert actually produced the best verified result? Now you can estimate an upper bound:
oracle expert selection Compare:
router-selected success vs oracle-selected success If oracle routing is only slightly better than one model, specialization itself has limited value.
If oracle routing is dramatically better but your router performs poorly, the bottleneck is routing.
This separates two questions beautifully:
Does specialization help? and Can we route to it reliably? Router Regret We can define a simple routing regret concept:
oracle outcome - selected-route outcome For score-based tasks:
routing_regret = oracle_score - selected_score For binary verified success, track how often:
a successful expert existed but the router chose a failing expert That is one of the most useful routing metrics you can collect.
Hard Cases Should Generate Training Evidence When routing fails:
store the task store the selected route store candidate expert outcomes store verification store cost/latency Now routing errors become future evidence.
That is how a static router can eventually become an adaptive one.
But that is a later mechanism.
Memory of outcomes is not yet learning.
Do Not Confuse Memory With Adaptive Routing A system may retrieve:
"Last time this kind of error went to the DB specialist." That is memory.
If the routing policy itself changes because repeated outcomes demonstrate that the DB specialist works better, that is learning/adaptation.
We will return to that distinction later in this series.
MoE and Search Can Be Combined The architectures in this series are composable.
For example:
MCTS node ↓ router ├→ code expert ├→ research expert └→ verifier Different branches could use different experts.
Or:
router ↓ expert ↓ Tree of Thoughts inside expert But composition multiplies cost and failure surfaces quickly.
Build one mechanism at a time.
Benchmark every escalation.
MoE and Self-Consistency Can Be Combined Instead of sampling one model five times:
same model × 5 we might sample specialists:
code model frontier generalist local model static analyzer That can produce more functional diversity.
But aggregation still needs strong evidence.
Expert disagreement does not become truth through voting.
MoE and Verification Are Natural Partners A powerful architecture is:
cheap specialist ↓ verification ├─ PASS → finish └─ FAIL/UNKNOWN ↓ stronger expert ↓ verification This creates an evidence-driven compute cascade.
Instead of predicting difficulty perfectly in advance, the runtime uses failed verification as evidence that more compute is required.
That is often easier to engineer than a perfect complexity classifier.
Safety and Permissions Belong in Expert Definitions Experts should not all have the same permissions.
Example:
@dataclass class ExpertPolicy: expert: str allowed_tools: set[str] can_write: bool can_deploy: bool can_access_secrets: bool A research expert might have:
read/search only A deployment expert may have:
production actions Routing therefore affects authorization.
The runtime must validate both:
Is this expert suitable? and Is this expert allowed to perform this action? Never Let the Router Invent Authority Even if the router chooses:
deployment_expert that does not automatically grant production access.
Authorization must be independent.
Architecture:
router proposes expert ↓ authorization layer ↓ allowed capability subset ↓ execution The model does not get to increase its own permissions by changing the route.
Production Debugging Checklist If your routed agent system performs badly, inspect it in this order.
1. Does specialization actually help? Run oracle routing.
If every expert performs similarly, the architecture may not need specialists.
2. Is the router choosing the wrong expert? Inspect confusion matrix and routing regret.
3. Are expert boundaries overlapping? Merge or sharpen ambiguous experts.
4. Is one expert receiving almost everything? Inspect expert collapse, but do not force balance without evidence.
5. Are cheap experts failing and escalating constantly? The cheap tier may be too weak or the route threshold too permissive.
6. Are expensive experts called unnecessarily? Tighten escalation and deterministic routes.
7. Do specialists produce unique wins? Ablate each expert.
8. Is the router using stale capability metadata? Version expert specs.
9. Are verification failures being mistaken for routing failures? Separate subsystem telemetry.
10. Does the whole mixture beat one strong model? If not, simplify.
A Controlled Experiment Suppose we are building a coding assistant.
Benchmark these architectures:
A: frontier model for every task B: local model for every task C: rule router → local/code/frontier D: learned/LLM router → specialists E: C + verification-driven escalation Record:
verified task success routing regret oracle routing success local-model utilization frontier escalation rate model calls latency cost Then introduce failure buckets:
simple Q&A repository retrieval bug diagnosis patch generation test repair architecture review Now you can see where specialization actually helps.
The Result We Want The goal is not:
more agents It is closer to:
cheap mechanism for easy work specialist mechanism for specialized work frontier model for genuinely hard work deterministic tools where deterministic tools are stronger verification after every consequential outcome That is a much more useful interpretation of mixture of experts.
A Practical Escalation Ladder Start here:
one model If cost is the problem:
cheap model ↓ verification-driven frontier escalation If domain failures are concentrated:
router ├→ domain specialist A ├→ domain specialist B └→ generalist If route uncertainty matters:
top-k experts ↓ strong aggregation / verification If routing itself becomes the bottleneck:
learned router Only add the next stage when the previous architecture produces evidence that it is needed.
The Core Principle The essential insight is simple:
A mixture of experts is not a collection of agents. It is a policy for assigning work to the capability most likely to solve it under your cost, latency, safety and verification constraints.
That policy can be:
deterministic, model-based, learned, state-dependent, failure-aware, cost-aware, or verification-driven. The architecture becomes useful when specialization creates real differences in outcome.
If every expert behaves the same, you do not have a meaningful mixture.
If routing adds more failure than specialization removes, simplify.
If one cheap expert solves almost everything, let it.
And if only a small fraction of requests truly require frontier-level inference, do not make every request pay for it.
Where We Are in the Series We now have:
00 When do advanced agent architectures help? ↓ 01 Chain of thought as intermediate computation ↓ 02 Self-consistency and disagreement ↓ 03 Tree of Thoughts ↓ 04 Monte Carlo Tree Search ↓ 05 Agent-level Mixture of Experts So far we have explored two major advanced-agent dimensions:
search and routing / specialization The next step is to combine specialized responsibilities into a deliberate control architecture.
That means moving beyond one router selecting one expert and asking:
What if planning, execution, criticism and verification should be separate roles with explicit contracts?
That is the subject of the next post:
Advanced Agents From First Principles 06: Does One Agent Plan, Execute and Judge Its Own Work? Build a Planner–Executor–Critic Architecture.