Are You Combining Every Agent Technique Into One Monster? Build a Mixture-of-Agents Runtime
You have a working agent.
Then you add retrieval.
Then memory.
Then Best-of-N.
Then critique and revision.
Then Tree of Thoughts.
Then MCTS.
Then specialist models.
Then adversarial review.
Then a planner, executor, critic and verifier.
Then a stronger model for hard cases.
Eventually the architecture starts to look like this:
request
↓
planner
↓
retrieval
↓
reasoning
↓
Best-of-N
↓
critic
↓
Tree of Thoughts
↓
MCTS
↓
frontier model
↓
second critic
↓
verifier
Every mechanism may have been individually reasonable.
The combined system can still be terrible.
It is slow.
It is expensive.
It is difficult to debug.
It is difficult to know which component actually helped.
And most importantly:
every request pays for every capability whether it needs it or not.
The answer is not to build an even larger agent.
The answer is to stop treating advanced techniques as permanent layers.
Treat them as alternative strategies.
Then route between those strategies.
That gives us an agent-level architecture that is closer to a Mixture of Agents:
┌→ direct model
├→ deterministic workflow
├→ retrieval + answer
request → strategy router ├→ Best-of-N
├→ critique + revise
├→ Tree of Thoughts
├→ MCTS
├→ specialist runtime
├→ adversarial review
└→ human escalation
↓
verification
The important idea is not the name.
The important idea is this:
Do not combine every reasoning mechanism into one agent. Route the task to the smallest strategy that can solve it reliably.
This is the architectural synthesis of the advanced-agent techniques we have built so far.
The Problem: Architecture Accretion
Agent systems often become complicated one feature at a time.
A production failure appears.
Someone adds another mechanism.
failure
↓
new mechanism
↓
new failure
↓
new mechanism
The individual decisions can all make sense.
The failure is that the mechanisms become stacked rather than selected.
Suppose a simple repository question can be answered by reading one file.
A badly accreted system might still perform:
- planning,
- retrieval,
- multiple reasoning samples,
- critic review,
- plan revision,
- specialist routing,
- final verification.
The task may cost ten times more without becoming meaningfully more reliable.
The system has confused:
capabilities available to the runtime
with:
capabilities that must execute on every request
Those are completely different things.
From Mixture of Experts to Mixture of Agents
In the previous routing post we built an agent-level Mixture of Experts.
That architecture routed work between different capabilities:
task
↓
router
├→ local model
├→ frontier model
├→ code specialist
├→ retriever
├→ deterministic tool
└→ verifier
A Mixture-of-Agents runtime moves the routing boundary upward.
Instead of routing only between experts, it can route between entire control strategies.
For example:
Strategy: DIRECT
model → answer → verify
or:
Strategy: RETRIEVAL
retrieve → answer → verify
or:
Strategy: SEARCH
branch → evaluate → prune → continue → verify
or:
Strategy: MCTS
select → expand → evaluate → backpropagate → verify
or:
Strategy: REVIEW
proposal → adversarial critics → evidence checks → adjudicate
These are not merely different prompts.
They are different runtime policies for spending computation.
The Core Abstraction: A Strategy
A useful strategy interface is surprisingly small.
from dataclasses import dataclass
from typing import Protocol, Any
@dataclass
class StrategyResult:
output: Any
status: str
evidence: list[dict]
cost: float
latency_ms: int
calls: int
class AgentStrategy(Protocol):
name: str
def supports(self, state: dict) -> bool:
...
def estimate_cost(self, state: dict) -> float:
...
def run(self, state: dict) -> StrategyResult:
...
The strategy does not need to know about every other strategy.
It only needs a contract:
input state
↓
strategy
↓
structured result + evidence + resource usage
That gives the runtime something it can compare.
A Strategy Registry
Now we can register available strategies.
class StrategyRegistry:
def __init__(self):
self._strategies = {}
def register(self, strategy: AgentStrategy):
self._strategies[strategy.name] = strategy
def available(self, state: dict):
return [
strategy
for strategy in self._strategies.values()
if strategy.supports(state)
]
A real registry might contain:
DIRECT
DETERMINISTIC_WORKFLOW
RETRIEVAL
BEST_OF_N
CRITIQUE_REVISE
TREE_OF_THOUGHTS
MCTS
SPECIALIST_ROUTING
PLANNER_EXECUTOR_CRITIC
ADVERSARIAL_REVIEW
HUMAN_ESCALATION
The important difference is architectural:
these mechanisms are alternatives available to the controller, not mandatory stages.
What Should the Router Look At?
A strategy router needs state.
Not just the user prompt.
Useful routing signals include:
task type
risk
current verification status
available evidence
uncertainty
previous failures
remaining budget
side-effect severity
latency target
required precision
search-space size
known deterministic path
Represent them explicitly.
@dataclass
class StrategyContext:
task_type: str
risk: float
uncertainty: float
previous_failures: int
verification_status: str
remaining_budget: float
deterministic_path_known: bool
irreversible_action: bool
Then routing becomes a decision over system state.
Start With a Deterministic Router
Do not immediately train another model to route your agents.
Start with rules.
def choose_strategy(ctx: StrategyContext) -> str:
if ctx.deterministic_path_known:
return "DETERMINISTIC_WORKFLOW"
if ctx.risk < 0.2 and ctx.uncertainty < 0.2:
return "DIRECT"
if ctx.task_type == "fact_lookup":
return "RETRIEVAL"
if ctx.previous_failures == 0 and ctx.uncertainty < 0.5:
return "CRITIQUE_REVISE"
if ctx.previous_failures >= 1 and ctx.uncertainty < 0.8:
return "TREE_OF_THOUGHTS"
if ctx.remaining_budget > 5.0:
return "MCTS"
return "HUMAN_ESCALATION"
This is intentionally crude.
That is useful.
You can inspect it.
You can measure it.
You can discover where it fails.
Only then do you have evidence that a learned router may be worth building.
Strategy Routing Is Not Model Routing
These are different decisions.
Model routing
task
↓
local model / frontier model / specialist model
Strategy routing
task state
↓
direct / retrieval / revision / search / MCTS / review
A strategy can itself contain model routing.
For example:
TREE_OF_THOUGHTS strategy
↓
local model generates branches
↓
learned scorer ranks branches
↓
frontier model handles difficult branch
↓
external verifier checks result
This produces a hierarchy:
strategy router
↓
strategy
↓
expert/model/tool router
Do not collapse those layers unless the simpler representation actually works better.
The Simplest Strategy Should Be a First-Class Expert
Advanced systems often make a subtle mistake.
They include many sophisticated routes but forget to include:
just answer the question
That should be a real strategy.
class DirectStrategy:
name = "DIRECT"
def supports(self, state):
return True
def estimate_cost(self, state):
return 0.01
def run(self, state):
output = call_model(state["prompt"])
return StrategyResult(
output=output,
status="PROPOSED",
evidence=[],
cost=0.01,
latency_ms=300,
calls=1,
)
Why?
Because the direct strategy is the baseline every advanced strategy must beat.
If Tree of Thoughts takes twenty calls and improves verified success from 96% to 96.2%, that may be a terrible trade.
Strategy Selection Needs Verification
A router can be perfectly confident and still choose the wrong strategy.
So the runtime should not assume:
route selected
=
task solved
Instead:
route selected
↓
strategy runs
↓
external verification
↓
PASS / FAIL / UNKNOWN
The verifier produces the feedback that allows strategy selection to become adaptive.
Failure Should Change Strategy, Not Just Repeat It
A common agent anti-pattern is:
strategy fails
↓
run same strategy again
↓
run same strategy again
That is retry, not adaptation.
A better controller asks:
what failure occurred?
Then changes strategy.
For example:
DIRECT failed because evidence missing
↓
RETRIEVAL
RETRIEVAL failed because sources conflict
↓
ADVERSARIAL_REVIEW
DIRECT produced several plausible implementations
↓
BEST_OF_N
Best-of-N candidates all fail tests differently
↓
TREE_OF_THOUGHTS
Tree search has delayed payoff and weak early scores
↓
MCTS
The escalation is failure-specific.
A Runtime Controller
A simple controller might look like this:
class MixtureOfAgentsRuntime:
def __init__(self, registry, router, verifier):
self.registry = registry
self.router = router
self.verifier = verifier
def run(self, state: dict):
history = []
while True:
strategy = self.router.choose(state, history)
result = strategy.run(state)
verification = self.verifier.verify(state, result)
history.append({
"strategy": strategy.name,
"result": result,
"verification": verification,
})
if verification.status == "PASS":
return result, history
if verification.status == "FAIL":
state = update_state_from_failure(
state,
verification,
)
if verification.status == "UNKNOWN":
state = request_more_evidence(
state,
verification,
)
if budget_exhausted(state, history):
return None, history
Notice what is missing.
There is no giant universal agent prompt.
The runtime controls composition.
Dynamic Composition
Sometimes one strategy is not enough.
The router may compose strategies sequentially.
For example:
RETRIEVAL
↓
DIRECT
↓
VERIFY
or:
TREE_OF_THOUGHTS
↓
BEST_OF_N on final branches
↓
VERIFY
or:
PLANNER_EXECUTOR
↓
ADVERSARIAL_REVIEW
↓
VERIFY
The danger is that dynamic composition can recreate the original monster architecture.
So composition needs the same rule:
Every added stage must be triggered by a measured failure or uncertainty.
Do not compose mechanisms merely because they are available.
The Monster-Agent Failure Mode
Suppose every request always runs:
planner
+ retrieval
+ self-consistency
+ Tree of Thoughts
+ MCTS
+ two critics
+ frontier verifier
That architecture has almost no useful adaptation.
It is simply a very expensive fixed pipeline.
A mixture architecture should instead look like:
Easy request
→ DIRECT
Known deterministic task
→ WORKFLOW
Knowledge gap
→ RETRIEVAL
Output variance
→ SELF_CONSISTENCY
Premature reasoning commitment
→ TREE_OF_THOUGHTS
Delayed-payoff search
→ MCTS
Specialist capability needed
→ EXPERT_ROUTING
High-risk unresolved claims
→ ADVERSARIAL_REVIEW
Different tasks should produce visibly different trajectories.
If they do not, your router is probably decorative.
Measure Strategy Utilization
Track how often each strategy is selected.
DIRECT 54%
RETRIEVAL 18%
CRITIQUE_REVISE 11%
TREE_OF_THOUGHTS 7%
SPECIALIST_ROUTING 5%
MCTS 2%
ADVERSARIAL_REVIEW 2%
HUMAN_ESCALATION 1%
The target is not equal utilization.
Equal utilization may actually be suspicious.
If MCTS is only justified for 2% of tasks, 2% may be exactly right.
The question is:
did each strategy earn the tasks it received?
Strategy Confusion Matrix
If you have labeled evaluation tasks, build a confusion matrix.
selected
D R T M A
actual DIRECT 81 4 3 0 2
actual RETR. 5 44 2 0 1
actual TOT 3 1 21 4 2
actual MCTS 0 0 5 12 1
This reveals systematic routing mistakes.
For example:
MCTS tasks → Tree of Thoughts
may indicate the router cannot detect delayed reward.
Or:
DIRECT tasks → MCTS
may indicate severe over-escalation.
Oracle Strategy Selection
A powerful offline diagnostic is oracle strategy selection.
For each benchmark task, run every eligible strategy.
Record which ones produce a verified success.
Task 17
DIRECT FAIL
RETRIEVAL PASS
BEST_OF_N PASS
TREE_OF_THOUGHTS PASS
MCTS PASS
The cheapest successful strategy is RETRIEVAL.
That gives us an oracle target:
oracle strategy = RETRIEVAL
Now compare the router.
If it chose MCTS, the task succeeded but the router wasted compute.
If it chose DIRECT and stopped after failure, the router missed an available successful strategy.
Strategy Regret
Define strategy regret as the difference between the selected strategy and the best available strategy under the objective.
The objective might include:
verified success
cost
latency
risk
For example:
utility = (
verified_success * 100
- cost * 4
- latency_seconds * 0.2
)
Then:
strategy regret
=
oracle utility - selected utility
This is much more useful than raw routing accuracy.
A router that selects a slightly different but equally successful cheap strategy may be perfectly acceptable.
Success Per Unit Compute
The main production metric should not be:
most sophisticated strategy selected
It should be something closer to:
verified successful tasks
-------------------------
compute / cost / latency
For example:
| Strategy | Verified success | Avg cost | Avg latency |
|---|---|---|---|
| Direct | 82% | $0.01 | 0.5s |
| Retrieval | 91% | $0.02 | 0.8s |
| Best-of-N | 94% | $0.06 | 2.2s |
| ToT | 95% | $0.18 | 6.5s |
| MCTS | 96% | $0.70 | 21s |
The conclusion is not:
MCTS wins
The conclusion may be:
MCTS belongs only on the small set of tasks where cheaper strategies fail.
That is exactly what the router is for.
Latency Budgets Matter
A strategy may be technically superior and operationally unusable.
For an interactive coding assistant:
500 ms
and:
30 seconds
are not interchangeable.
So routing constraints should include latency.
@dataclass
class StrategyBudget:
max_cost: float
max_latency_ms: int
max_calls: int
max_search_nodes: int
A strategy that cannot fit inside the remaining budget should not be eligible.
Risk Changes Routing
Difficulty and risk are different dimensions.
A simple action can be high risk.
Example:
delete production database
The reasoning may be trivial.
The verification and authorization requirements are not.
So strategy selection should consider:
difficulty
uncertainty
risk
separately.
A low-difficulty/high-risk task might route to:
deterministic workflow
+
strong authorization
+
independent verifier
rather than to a complex reasoning strategy.
Authorization Is Not a Strategy Choice
The router may choose a powerful executor.
That must not grant additional permissions.
router chooses deployment expert
does not imply:
deployment expert may deploy to production
Authorization remains an independent runtime boundary.
strategy selection
↓
capability request
↓
authorization check
↓
execution
Never let a model route itself into broader authority.
Coding Agents
A coding agent is an excellent example of strategy routing.
Direct
Use for:
explain this function
rename this local variable
write a small pure helper
Retrieval
Use when repository knowledge is missing:
where is authentication configured?
which module owns migrations?
Deterministic workflow
Use for known mechanical changes:
format
lint
run tests
update generated files
Best-of-N
Use when several candidate implementations are cheap to generate and test.
Tree of Thoughts
Use when multiple architectural directions should remain alive.
MCTS
Use only when early implementation quality is a weak predictor of eventual test/integration success and search statistics can be reused.
Adversarial review
Use for security, regression risk, API compatibility and architecture-boundary checks.
The repository state remains external truth throughout.
Research Agents
Research strategy routing might look like:
known factual lookup
→ retrieval
single clear primary source
→ retrieval + extraction
conflicting sources
→ adversarial evidence review
multiple plausible hypotheses
→ Tree of Thoughts
long multi-stage investigation
→ planner-executor
high disagreement after evidence
→ specialist adjudication
The point is not to make every research task a debate.
Most factual lookups should stay cheap.
Customer Support
Support agents benefit enormously from not treating all tickets equally.
password reset
→ deterministic workflow
order status
→ database/tool lookup
policy question
→ retrieval
ambiguous billing dispute
→ specialist route
high-value account escalation
→ human
A giant reasoning agent is usually worse than a well-routed support system.
Data and Analytics Agents
Different strategies fit different analytical tasks.
simple arithmetic
→ deterministic calculator
known SQL query pattern
→ deterministic query builder
schema discovery
→ retrieval/catalog tools
ambiguous metric definition
→ reasoning + evidence
several competing causal explanations
→ hypothesis search
high-impact conclusion
→ adversarial review + verification
Again, not everything should become multi-agent reasoning.
Browser Automation
Browser agents have especially strong reasons to route strategies carefully because actions can have side effects.
read page
→ browser tool
extract table
→ deterministic parser
navigate known flow
→ workflow
unknown site structure
→ planning/search
purchase / submit / delete
→ authorization + explicit verification
Search strategies should operate over read-only or reversible actions where possible.
DevOps and Incident Response
A DevOps mixture might contain:
log retriever
metrics analyzer
runbook workflow
root-cause hypothesis search
rollback planner
specialist reviewer
human incident commander
Routing should use live evidence:
alerts
logs
metrics
recent deployments
service topology
current incident state
Not just a textual description of the incident.
Router Confidence Is Not Success Confidence
These are separate signals.
router confidence
=
how sure are we this strategy is appropriate?
success confidence
=
how much evidence supports task completion?
A router might be 99% sure that MCTS is the right strategy.
The MCTS result can still fail verification.
Do not collapse the two.
Strategy Routing Can Learn
The previous post introduced learning from verified trajectories.
Strategy routing is an obvious policy surface.
A trajectory can record:
task features
selected strategy
alternative strategies tested offline
verification outcome
cost
latency
failure class
From this we can learn candidate routing rules.
For example:
When repository tests fail after two independent direct patches,
Tree of Thoughts has 31% higher verified success than repeated revision.
That is a candidate lesson.
It still needs evaluation before promotion.
Do not let one successful MCTS run rewrite the router.
Shadow Routing
A safe way to improve the router is shadow evaluation.
Production uses:
router_v4
while a candidate router computes:
router_v5_shadow
without controlling execution.
Compare:
selected strategy
oracle strategy
verified outcome
estimated cost
actual cost
Only then promote the new router.
Strategy-Level Failure Attribution
Once strategies are explicit, failures become easier to classify.
Routing failure
A better strategy existed but was not selected.
Strategy failure
The selected strategy was appropriate but executed poorly.
Expert failure
The strategy selected the wrong model/tool/specialist internally.
Budget failure
The correct strategy was selected but lacked enough resources.
Verification failure
The strategy succeeded but the verifier rejected it, or failed but the verifier passed it.
Authorization failure
A strategy attempted a capability it was not allowed to use.
Learning failure
Past verified evidence should have changed routing policy but did not, or a bad rule was promoted.
This separation is extremely valuable in production.
Do Not Build a Meta-Agent That Merely Thinks About Agents
A common temptation is:
user request
↓
meta-agent
↓
"which agent should I use?"
That can work.
But if the meta-agent is simply another unconstrained LLM call, you may have moved the problem instead of solving it.
Start with observable routing features and deterministic rules.
Add model-based strategy selection only when semantic classification genuinely helps.
Then measure its routing regret.
A Better Escalation Ladder
A useful production sequence is:
1. deterministic path if known
2. direct model
3. retrieval / tool evidence
4. revision or Best-of-N
5. specialist routing
6. Tree of Thoughts
7. adversarial review
8. MCTS
9. stronger model / larger budget
10. human escalation
This is not a universal ordering.
The correct ordering depends on the task.
But the principle is useful:
cheap, testable mechanisms first
expensive, speculative mechanisms later
Benchmark the Runtime, Not Just the Strategies
Individual strategy benchmarks are not enough.
The mixture itself needs evaluation.
Compare:
Baseline A
frontier model for everything
Baseline B
fixed advanced stack
Baseline C
deterministic rule router
Baseline D
adaptive router with verification feedback
Baseline E
learned router trained on verified trajectories
Measure:
verified success
cost
latency
strategy regret
unnecessary escalation rate
missed escalation rate
human escalation rate
failure recovery rate
That is the real test.
Important Metrics
Verified task success
The only success metric that ultimately matters.
Strategy utilization
How often each strategy runs.
Strategy regret
How much utility was lost relative to the best available strategy.
Unnecessary escalation rate
advanced strategy selected
when cheaper strategy would have verified successfully
Missed escalation rate
cheap strategy failed
and advanced strategy would have succeeded
but was not selected
Recovery rate
How often strategy switching turns a failed attempt into verified success.
Cost per verified success
The real economic denominator.
Latency per verified success
Critical for interactive systems.
Strategy-switch count
Too many switches may indicate thrashing.
Router calibration
Does route confidence correspond to actual route quality?
Strategy Thrashing
Adaptive systems can oscillate.
DIRECT
→ RETRIEVAL
→ DIRECT
→ RETRIEVAL
→ DIRECT
Or:
TREE_OF_THOUGHTS
→ MCTS
→ TREE_OF_THOUGHTS
That is strategy thrashing.
Prevent it with:
strategy history
failure fingerprints
switch limits
no-progress windows
budget limits
The router should know what has already been tried.
The Runtime Should Preserve Shared State
Strategies should not communicate by repeatedly summarizing the entire task in prose.
Use shared structured state.
@dataclass
class AgentState:
goal: str
constraints: list[str]
observations: list[dict]
evidence: list[dict]
failures: list[dict]
attempts: list[dict]
budget_remaining: float
Then:
DIRECT fails
and:
TREE_OF_THOUGHTS starts
without losing the evidence already collected.
This avoids handoff loss across strategies.
Do Not Let Strategies Rewrite the Goal
The user goal should remain immutable unless the user changes it.
Strategies may propose interpretations.
They may not silently redefine success.
original goal
↓
immutable goal contract
↓
strategy-specific plan
The verifier checks against the original goal contract.
Not against whatever interpretation made the selected strategy look successful.
The Final Architecture
We can now describe a fairly complete advanced runtime.
┌─────────────────────┐
│ Goal Contract │
└──────────┬──────────┘
↓
┌─────────────────────┐
│ Structured State │
└──────────┬──────────┘
↓
┌─────────────────────┐
│ Strategy Router │
└──────────┬──────────┘
↓
┌────────────────────────────────────────────────┐
│ │
↓ ↓ ↓ ↓ ↓
Direct Retrieval Search MCTS Review
│ │ │ │ │
└────────────┴─────────────┴────────────┴───────┘
↓
┌─────────────────────┐
│ External Verification│
└──────────┬──────────┘
↓
PASS / FAIL / UNKNOWN / ESCALATE
↓
┌─────────────────────┐
│ Trajectory Evidence │
└──────────┬──────────┘
↓
offline learning loop
This is not one giant intelligent agent.
It is a runtime that coordinates several bounded computational strategies.
That distinction matters.
The Deeper Pattern
Across this entire series, the advanced techniques can now be seen as different answers to one question:
Where should the next unit of computation go?
Chain of Thought:
into another reasoning step
Self-Consistency:
into another sample
Tree of Thoughts:
into another branch
MCTS:
into the branch whose exploration/exploitation value is highest
Mixture of Experts:
into the specialist with the right capability
Planner–Executor–Critic:
into the role responsible for the current failure
Adversarial Review:
into testing the most important unresolved claim
Adaptive Agents:
into whichever mechanism the current evidence justifies
Learning:
into improving future allocation policy
Mixture of Agents:
into the entire reasoning strategy with the highest expected utility
That is the architectural endpoint.
Do You Actually Need a Mixture of Agents?
Probably not at first.
Start here:
model
↓
structured action
↓
tool
↓
verification
Only add strategy routing when you can show that different task classes genuinely benefit from different computational policies.
The evidence should look something like:
Task class A
DIRECT: 98% verified success
MCTS: 98% verified success, 25x cost
Task class B
DIRECT: 52%
TREE_OF_THOUGHTS: 81%
MCTS: 84%
Task class C
DIRECT: 70%
RETRIEVAL: 96%
Now routing has a reason to exist.
Without that evidence, a Mixture-of-Agents architecture may simply be another abstraction layer.
Production Checklist
Before shipping strategy routing, ask:
- Are strategies explicit runtime units rather than vague prompt roles?
- Is the direct/simple path a first-class strategy?
- Are routing features observable?
- Can deterministic rules solve routing adequately?
- Is every strategy externally verified?
- Are cost and latency part of routing?
- Are risk and difficulty modeled separately?
- Is authorization independent from strategy choice?
- Can failures trigger a different strategy rather than blind retry?
- Do strategies share structured state?
- Is the original goal immutable?
- Can you calculate oracle strategy performance offline?
- Can you measure strategy regret?
- Can you detect unnecessary escalation?
- Can you detect missed escalation?
- Can you detect strategy thrashing?
- Are strategy changes versioned and reversible?
- Does the mixture beat a fixed frontier-model baseline?
- Does it beat a fixed advanced-stack baseline?
If not, the architecture is probably not ready.
Final Rule
A powerful agent platform should not ask:
How many advanced techniques can we combine?
It should ask:
What is the cheapest computational strategy that can produce a verified result for this task state?
That changes the architecture completely.
The runtime stops being a giant stack of intelligence-shaped components.
It becomes a policy for allocating computation under evidence, cost, latency and risk constraints.
And that is the real point of a Mixture of Agents.
Do not build one monster agent. Build a runtime that knows which kind of agentic computation to spend next — and make every escalation earn its cost with verified evidence.
The next step is to stop looking at mechanisms one at a time and evaluate the system as a whole: how do you benchmark an advanced agent architecture without letting demos, model judges or cherry-picked tasks fool you?
