# Advanced Agents From First Principles 04: Does Your Agent Prune Good Ideas Too Early? Use Monte Carlo Tree Search for Long-Horizon Reasoning
# Book: Advanced Agents From First Principles
# Chapter URL: https://programmer.ie/books/advanced-agents-from-first-principles/04-chapter/
# Prose fingerprint (sha256, normalized): 79c82d04ce2197966afc9a9d7340b4900451607e2fd990fab3bab91237f59c56
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

A common failure in search-based agents is easy to miss.
The agent generates several plausible branches.
It scores them.
One branch looks weak.
So the runtime prunes it.
Later, you discover that the discarded branch was the only one that could have reached the correct solution.
The problem was not generation.
The problem was not necessarily the model.
The problem was search allocation.
The agent spent too much compute exploiting what looked good early and too little compute exploring alternatives whose value only became visible later.
That is the kind of problem Monte Carlo Tree Search is designed to address.
This article builds MCTS from first principles and asks the question that matters in production:
When does MCTS actually solve a problem that simpler search does not?
The answer is not “when the task is hard.”
The answer is closer to:
Use MCTS when early evaluations are unreliable, consequences unfold over multiple steps, and additional exploration can reveal delayed value that greedy pruning would otherwise destroy.
That distinction matters.
Because MCTS can also be expensive, noisy, slow, and completely unnecessary.
The Problem: Early Scores Can Lie In the previous post we built Tree of Thoughts as search over intermediate semantic states.
The simplest version looked like this:
state ↓ branch ↓ evaluate ↓ keep top K ↓ branch again That is often implemented as beam-style search.
At every depth, the best-looking partial states survive.
That works well when the evaluator is reasonably predictive of eventual success.
But suppose the search space looks like this:
root ├── A score 0.90 │ └── A1 score 0.88 │ └── terminal FAIL │ ├── B score 0.74 │ └── B1 score 0.80 │ └── terminal PASS │ └── C score 0.68 └── C1 score 0.70 └── terminal FAIL A beam width of one chooses A.
A shallow heuristic says A is clearly best.
But the terminal evidence says B was the correct route.
The search algorithm has confused:
looks promising now with:
produces good outcomes later MCTS exists to manage that uncertainty.
Search Is Compute Allocation It helps to stop thinking of MCTS as a mysterious game-playing algorithm.
At the agent level, MCTS is a way of deciding:
Which partial trajectory deserves the next unit of compute?
Suppose you have 100 model calls available.
You could spend them evenly:
A: 25 B: 25 C: 25 D: 25 Or greedily:
A: 70 B: 10 C: 10 D: 10 Or adaptively:
A initially looks strongest ↓ explore A ↓ A's deeper outcomes disappoint ↓ redirect compute toward B ↓ B begins producing stronger terminal evidence ↓ allocate more compute to B That last pattern is the heart of MCTS.
It continuously balances two competing objectives:
exploit what currently looks good and:
explore what remains uncertain The Four Operations A basic MCTS loop contains four operations:
1. selection 2. expansion 3. simulation / evaluation 4. backpropagation Then it repeats.
root ↓ selection ↓ leaf ↓ expansion ↓ new child ↓ simulation / evaluation ↓ reward ↓ backpropagation ↓ update tree statistics ↓ repeat Let’s build each part.
1. Selection Selection chooses the existing tree node that should receive more search effort.
If we always choose the node with the highest average reward, we become greedy.
That creates a familiar failure:
first lucky branch ↓ looks best ↓ gets more samples ↓ other branches remain underexplored We need a mechanism that rewards both:
high estimated value insufficient exploration A common form is UCB1.
For child node i:
UCB(i) = mean_reward(i) + c * sqrt(log(parent_visits) / child_visits(i)) The first term is exploitation.
mean_reward(i) The second term is exploration.
sqrt(log(parent_visits) / child_visits(i)) The constant c controls how aggressively we explore.
Large c:
more exploration Small c:
more exploitation The important thing is not the exact equation.
The important thing is the policy:
A branch that looks good should receive more compute, but a branch that has barely been tested should not be discarded merely because its current estimate is uncertain.
A Tiny UCB Function from __future__ import annotations import math def ucb_score( mean_reward: float, parent_visits: int, child_visits: int, exploration: float = 1.4, ) -> float: if child_visits == 0: return float("inf") return mean_reward + exploration * math.sqrt( math.log(max(parent_visits, 1)) / child_visits ) Unvisited children get infinite priority.
That forces the runtime to gather at least some evidence before assuming they are worthless.
2. Expansion Once selection reaches a node that can be expanded, the agent generates one or more possible next states.
For a coding agent:
current repository state ├── inspect failing test ├── inspect implementation ├── inspect recent git diff └── reproduce failure locally For incident response:
current incident state ├── inspect application logs ├── inspect database latency ├── inspect queue depth └── inspect deployment history For research:
current hypothesis ├── search primary source A ├── search dataset B ├── test alternative explanation C └── inspect contradiction D Expansion is normally model-driven.
But it does not have to be.
The available actions might come from a deterministic action space.
That is often better when the domain already has clearly defined operations.
3. Simulation or Evaluation Classic MCTS often performs a rollout from the newly expanded node until a terminal outcome is reached.
For an agent, that may be expensive.
A rollout could mean:
multiple LLM calls tool execution running tests querying external systems simulated actions invoking a cheaper value model So in practice we usually have several options.
Full rollout partial state ↓ continue until terminal state ↓ measure terminal reward Strong evidence.
Potentially expensive.
Partial rollout partial state ↓ advance 2-3 steps ↓ evaluate resulting state Cheaper.
Less reliable.
Value model partial state ↓ learned scorer estimates eventual success Very cheap.
But now MCTS depends heavily on value-model calibration.
Environment proxy candidate patch ↓ run targeted tests ↓ reward from failing-test reduction Often excellent when available.
For agent systems, this is crucial:
Use real environment evidence whenever the environment can cheaply reveal progress.
Do not replace a test suite with an LLM saying the patch “looks likely to work.”
4. Backpropagation After evaluating a node, MCTS propagates the resulting reward back through the path that produced it.
Suppose the selected path was:
root ↓ B ↓ B3 ↓ B3a And the rollout reward was:
0.9 Then every node in the path updates its statistics.
For example:
node.visits += 1 node.value_sum += reward Its mean value becomes:
mean = node.value_sum / node.visits The important consequence is that early nodes slowly accumulate evidence about what their descendants tend to produce.
That is why MCTS can recover from misleading shallow evaluations.
A Minimal Node from dataclasses import dataclass, field from typing import Any @dataclass class SearchNode: state: Any parent: "SearchNode | None" = None action: str | None = None children: list["SearchNode"] = field(default_factory=list) visits: int = 0 value_sum: float = 0.0 terminal: bool = False @property def mean_value(self) -> float: if self.visits == 0: return 0.0 return self.value_sum / self.visits This node contains two different kinds of information.
The semantic state:
state And the search statistics:
visits value_sum mean_value That distinction becomes important later.
A Minimal MCTS Skeleton class MCTS: def __init__(self, exploration: float = 1.4): self.exploration = exploration def select_child(self, node: SearchNode) -> SearchNode: return max( node.children, key=lambda child: ucb_score( mean_reward=child.mean_value, parent_visits=node.visits, child_visits=child.visits, exploration=self.exploration, ), ) def backpropagate(self, node: SearchNode, reward: float) -> None: current = node while current is not None: current.visits += 1 current.value_sum += reward current = current.parent This is not yet a complete agent.
But the allocation mechanism is already visible.
Tree of Thoughts vs MCTS This distinction gets confused constantly.
A typical beam-style Tree of Thoughts system asks:
Which K branches look best right now? Then it keeps those branches.
MCTS asks something closer to:
Which branch should receive the next unit of search compute, considering both what we know and what we still do not know? That difference matters when value is delayed.
Beam-style search evaluate frontier ↓ keep top K ↓ discard rest Once a branch is pruned, it is gone.
MCTS retain tree statistics ↓ allocate visits adaptively ↓ underexplored branches remain eligible The branch does not have to look best immediately.
It has to justify further exploration relative to alternatives.
When MCTS Is Actually Useful MCTS is especially attractive when four conditions hold.
1. Early scores are weak predictors If a shallow score already predicts terminal success extremely well, beam search is simpler.
MCTS helps when:
partial-state score ↓ poorly predicts ↓ terminal outcome 2. The task has delayed consequences For example:
architectural change ↓ seems larger initially ↓ removes entire class of downstream failures A greedy search might prefer the smaller patch.
MCTS can keep allocating some compute to the more uncertain structural branch.
3. Branch statistics are reusable Repeated visits must tell us something useful.
If every rollout is unrelated noise, backpropagating averages does little.
4. Search is expensive enough that allocation matters If you can exhaustively evaluate every branch cheaply, you do not need MCTS.
Just evaluate everything.
When MCTS Is Overkill There are many cases where MCTS should not be your first answer.
If the task is:
one prompt ↓ one good answer use one model call.
If the task is:
generate 5 complete candidates ↓ verify each ↓ choose best use Best-of-N.
If the task is:
branch a few times ↓ partial score is reliable ↓ keep top K use beam search / Tree of Thoughts.
MCTS becomes interesting when:
branch quality is uncertain + reward arrives late + search budget is limited + repeated evidence improves value estimates The Most Important Question: What Is the Reward? A beautiful search policy cannot rescue a bad reward signal.
Suppose your coding agent rewards:
number of tests passed That may encourage:
remove failing tests or:
skip test execution or:
modify assertions depending on what actions the runtime permits.
The reward must represent the actual objective.
For example:
required tests pass + no regression tests fail + changed behavior matches specification + patch scope remains bounded The exact weights matter less than the principle:
The reward function is part of the agent specification.
Reward Shaping Can Help—and Mislead Terminal rewards are often sparse.
PASS = 1 FAIL = 0 That is clean but may provide little guidance.
We can add intermediate signals.
For a coding agent:
+0.2 target test failure reproduced +0.3 number of failing tests reduced +0.3 target test passes +0.2 full suite passes But reward shaping creates a new risk.
The agent may optimize the proxy instead of the goal.
So we should always retain a distinction between:
search reward and:
final verification The search reward guides compute.
The verifier decides whether the result is actually acceptable.
Search Reward Is Not Verification This distinction is so important that it is worth making explicit.
high MCTS value ≠ verified success A branch can accumulate excellent expected reward and still fail final acceptance.
Therefore the architecture should look like:
MCTS ↓ select candidate trajectory ↓ execute / materialize result ↓ external verifier ↓ PASS / FAIL / UNKNOWN The tree decides where to search.
The verifier decides whether reality supports the result.
Coding Agent Example Consider a coding agent debugging a failing integration test.
The root state contains:
repository state failing test error message current diff Initial branches might be:
A: patch failing function directly B: inspect call graph C: inspect recent schema change D: inspect test fixture A superficial score might rank A first.
It proposes the obvious fix.
But perhaps the real defect is a schema mismatch introduced two layers earlier.
MCTS can allocate some rollouts to B and C.
A rollout for B might become:
inspect call graph ↓ identify shared parser ↓ modify parser ↓ run targeted tests ↓ run regression suite The terminal evidence can then flow backward to B.
Over repeated visits, the search policy learns that B produces stronger outcomes than A even though A looked better initially.
Coding Agents: A Good MCTS Reward For code, the strongest reward often comes from the environment.
For example:
def coding_reward(result) -> float: score = 0.0 if result.reproduced_failure: score += 0.10 if result.target_test_passes: score += 0.40 if result.regression_suite_passes: score += 0.40 if result.diff_is_bounded: score += 0.10 return score But final acceptance still remains separate.
def verify_patch(result) -> bool: return ( result.target_test_passes and result.regression_suite_passes and result.requirements_satisfied ) Research Agent Example A research agent may have several competing explanations.
hypothesis A hypothesis B hypothesis C Early evidence may favor A.
But additional evidence could overturn it.
MCTS can treat evidence acquisition itself as part of the action space.
A ├── inspect source supporting A ├── inspect source contradicting A └── search for primary dataset This is much healthier than repeatedly asking an LLM:
How confident are you in hypothesis A? The search should spend compute reducing uncertainty through evidence.
Planning and Scheduling Scheduling problems are another natural fit.
A partial schedule may look mediocre because it reserves capacity.
But that capacity may prevent a major conflict later.
Greedy planning often prefers immediate utilization.
MCTS can evaluate longer-horizon consequences.
Examples include:
job-shop scheduling delivery routing resource allocation workflow orchestration project sequencing The important condition remains the same:
Early local quality is not a reliable proxy for eventual global quality.
Incident Response Incident-response agents also have delayed consequences.
Suppose the agent sees high database latency.
Possible actions:
restart database inspect query plan inspect recent deployment inspect connection pool Restarting might produce an immediate improvement.
That gives it a high short-term reward.
But if the root cause is a bad deployment, the problem returns.
A longer branch:
inspect deployment ↓ identify query regression ↓ rollback safely ↓ verify latency ↓ verify error rate may have better long-term value.
This is exactly the kind of delayed-payoff structure where MCTS can become useful.
Browser and UI Agents MCTS can also help browser agents when navigation choices create delayed constraints.
For example:
checkout ├── guest checkout ├── create account └── login existing account One path might appear shorter.
But later it may require information the agent does not have.
A more expensive early branch could reduce downstream uncertainty.
Again:
shortest immediate path ≠ best complete trajectory The Action Space Matters MCTS assumes the search tree contains useful actions.
If expansion generates nonsense, search cannot rescue it.
So we still need strong action-space design.
For every node ask:
What actions are actually legal here? not:
What random ideas can the model imagine? For a coding agent, legal actions might include:
read_file search_symbol run_test inspect_diff apply_patch The runtime can constrain expansion according to state.
if no patch exists: do not offer revert_patch This dramatically reduces branching noise.
Progressive Widening Large agent action spaces create another problem.
Suppose each node can produce 50 plausible children.
Expanding them all immediately is expensive.
Progressive widening addresses this by allowing the number of children to grow with node visits.
Conceptually:
few visits ↓ few children many visits ↓ allow more children That lets the search deepen promising nodes before exploding the branching factor.
A simple policy might be:
allowed_children = max(1, int(node.visits ** 0.5)) The exact schedule should be benchmarked.
The principle is what matters.
Value Noise LLM-based value functions can be extremely noisy.
Ask the same model to score the same partial branch repeatedly and you may get:
0.82 0.67 0.91 0.73 If MCTS treats each score as precise, the search statistics become misleading.
Useful diagnostics include:
value mean value variance rank stability confidence interval For high-noise evaluators, repeated evaluation may be justified.
But before adding more model calls, ask whether the environment offers a stronger signal.
Correlated Rollouts Ten rollouts are not necessarily ten independent pieces of evidence.
If every rollout uses:
the same model the same prompt the same retrieval context the same tool policy then their errors may be highly correlated.
You can have:
10 visits but much less than:
10 independent observations This is the same problem we saw with self-consistency.
So rollout diversity matters.
Possible variations include:
strategy-conditioned rollout policies different model tiers deterministic environment checks different retrieved evidence different tool routes But diversity should be purposeful.
Randomness alone is not evidence quality.
Exploration Constant Failure Modes The exploration coefficient controls how aggressively MCTS revisits uncertain branches.
Too low:
search becomes nearly greedy Too high:
search wastes compute on obviously weak branches Do not treat 1.4 as a magical constant just because it appears in examples.
Sweep it.
c = 0.25 c = 0.5 c = 1.0 c = 1.4 c = 2.0 Then measure verified outcomes, not tree aesthetics.
Search Budgets Must Be Explicit MCTS is dangerous without budgets because the algorithm always has another branch worth investigating.
Bound at least:
max iterations max tree nodes max model calls max tool calls max wall time max cost max depth For example:
@dataclass class SearchBudget: max_iterations: int = 100 max_nodes: int = 500 max_model_calls: int = 250 max_seconds: float = 60.0 max_cost_usd: float = 2.0 A production search system should have a named termination reason.
VERIFIED_SUCCESS ITERATION_BUDGET NODE_BUDGET TIME_BUDGET COST_BUDGET NO_PROGRESS NO_VALID_ACTIONS Early Stopping MCTS does not need to consume the entire budget.
Useful stopping conditions include:
verified terminal success or:
best branch value stable for N iterations or:
additional search produces no meaningful improvement or:
value gap between best and alternatives exceeds threshold But be careful.
Stopping because one branch’s model score looks dominant can recreate the exact premature-pruning problem MCTS was introduced to solve.
Whenever possible, use stronger evidence.
Do Not Search Irreversible Side Effects This rule from the earlier search post becomes even more important with MCTS.
You do not want rollouts that repeatedly:
send emails charge credit cards restart production services merge pull requests delete data Search should usually operate over:
plans simulated state sandboxed environments temporary worktrees read-only diagnostics reversible transformations Then select a trajectory.
Then execute the real side effect once.
search ↓ select ↓ verify preconditions ↓ commit irreversible action ↓ verify postconditions A More Complete MCTS Skeleton Here is a deliberately small implementation structure.
from dataclasses import dataclass from typing import Protocol class Environment(Protocol): def actions(self, state): ... def transition(self, state, action): ... def is_terminal(self, state) -> bool: ... def reward(self, state) -> float: ... @dataclass class MCTSConfig: iterations: int = 100 exploration: float = 1.4 max_depth: int = 8 class MCTSAgent: def __init__(self, env: Environment, config: MCTSConfig): self.env = env self.config = config def search(self, root: SearchNode) -> SearchNode: for _ in range(self.config.iterations): leaf = self._select(root) child = self._expand(leaf) reward = self._evaluate(child) self._backpropagate(child, reward) return self._best_root_child(root) Now the methods.
def _select(self, node: SearchNode) -> SearchNode: current = node while current.children and not current.terminal: current = max( current.children, key=lambda child: ucb_score( child.mean_value, current.visits, child.visits, self.config.exploration, ), ) return current Expansion:
def _expand(self, node: SearchNode) -> SearchNode: if node.terminal: return node actions = self.env.actions(node.state) if not actions: node.terminal = True return node action = actions[0] next_state = self.env.transition(node.state, action) child = SearchNode( state=next_state, parent=node, action=str(action), terminal=self.env.is_terminal(next_state), ) node.children.append(child) return child Evaluation:
def _evaluate(self, node: SearchNode) -> float: return self.env.reward(node.state) Backpropagation:
def _backpropagate(self, node: SearchNode, reward: float) -> None: current = node while current is not None: current.visits += 1 current.value_sum += reward current = current.parent And final selection:
def _best_root_child(self, root: SearchNode) -> SearchNode: return max(root.children, key=lambda child: child.visits) This is intentionally incomplete.
A serious implementation still needs:
unexpanded-action tracking rollout policies duplicate-state handling budgets provenance concurrency control state hashing failure handling verification But the architecture is visible.
Why Select by Visits at the End? During search we need exploration.
At the end, we do not.
So the final decision often uses:
most visited child or:
highest mean value child rather than UCB.
The exploration bonus is a search allocation device.
It should not automatically become the production decision rule.
Transpositions and Duplicate States Agent trees often contain different paths that reach the same state.
For example:
read tests → inspect implementation and:
inspect implementation → read tests may lead to the same knowledge state.
If we represent them as completely independent nodes, the tree duplicates work.
A state fingerprint can help.
import hashlib import json def state_fingerprint(state: dict) -> str: canonical = json.dumps(state, sort_keys=True, default=str) return hashlib.sha256(canonical.encode()).hexdigest() Then the runtime can detect repeated states.
This converts the search structure from a pure tree toward a graph.
That can substantially improve efficiency.
Provenance Is Mandatory Advanced search quickly becomes impossible to debug without lineage.
Every node should record at least:
node_id parent_id action state fingerprint visit count value mean value variance creation source model / tool used observations verification evidence Then when a surprising branch wins, you can reconstruct why.
Without provenance, MCTS becomes a stochastic black box wrapped around another stochastic black box.
That is not a production architecture.
Parallel MCTS Agent workloads often involve expensive model or tool calls.
Parallel search is tempting.
But naive parallelism creates a problem.
Several workers may all select the same currently promising node before any of them updates its statistics.
That duplicates effort.
A common technique is a virtual loss.
Temporarily penalize a node while a worker explores it.
worker selects node ↓ apply virtual loss ↓ other workers prefer alternatives ↓ worker finishes rollout ↓ remove virtual loss ↓ backpropagate real reward This is useful when model calls dominate latency.
But parallelism should still be benchmarked against cost and duplicated work.
Learned Value Models The model series now becomes directly useful again.
Instead of asking an LLM judge to score every node, we could use a learned value model.
Conceptually:
partial agent state ↓ encoder ↓ value model ↓ estimated probability of eventual success That value model might be simple.
For example:
MR.Q-style scalar scorer or recurrent if trajectory history matters.
But the evidence rule remains:
A value model earns its place only if it predicts eventual verified outcomes well enough to improve search allocation.
Do not use it because the architecture diagram looks sophisticated.
Calibrating the Value Model Suppose the scorer predicts:
0.8 for 100 states.
Roughly 80 should eventually succeed if the score is well calibrated as a probability.
Useful diagnostics include:
Brier score calibration curve rank correlation AUC precision among top-ranked branches But the most important metric remains downstream:
Does MCTS using this scorer improve verified task success? Failure Attribution When MCTS fails, do not say:
MCTS did not work Localize the failure.
No correct branch ever appears generator / action-space failure Correct branch appears but receives few visits selection / exploration failure Correct branch receives visits but low rewards evaluator / rollout failure Correct branch has good evidence but loses at root selection aggregation / final-selection failure Correct branch wins search but task still fails execution / verification failure That decomposition is extremely useful in production.
Search Metrics For every run, log:
iterations nodes created nodes revisited max depth branching factor rollouts model calls tool calls value evaluations unique states wall time cost termination reason verified outcome Then add search-quality metrics.
oracle generated success best verified branch discovered visit concentration value variance root action entropy pruning / allocation regret For example:
oracle discovered success = true selected branch verified success = false This immediately tells us the search generated a winning trajectory but failed to allocate/select correctly.
The Experiment That Justifies MCTS Do not compare MCTS against nothing.
Compare it against increasingly complex baselines.
A: one-shot agent B: Best-of-N C: Tree of Thoughts / beam width 2 D: Tree of Thoughts / beam width 4 E: MCTS 25 iterations F: MCTS 100 iterations Measure:
verified success oracle discovered success model calls tool calls latency cost cost per verified success Then ask:
Does adaptive exploration outperform simple frontier pruning? If not, keep the simpler system.
Example Benchmark Table architecture success calls latency cost --------------------------------------------------- one-shot 62% 1 1.0x 1.0x Best-of-5 71% 5 1.8x 5.0x beam width 3 78% 14 3.2x 14.0x MCTS 25 79% 23 4.5x 23.0x MCTS 100 80% 88 12.0x 88.0x If your real data looks like this, MCTS may not be worth it.
The gain from beam search to MCTS is tiny.
A better result might look like:
architecture success calls ---------------------------------- beam width 3 61% 14 MCTS 25 74% 23 Now the adaptive search policy may be solving a real delayed-value problem.
Application Matrix Software type Search state Delayed payoff example Strong reward signal Coding agent repository + hypotheses + diff larger architectural fix beats small local patch tests + regression suite Research agent hypotheses + evidence graph initially weak hypothesis gains primary-source support source-backed claim verification Incident response system observations + interventions root-cause fix beats temporary restart latency/error recovery + stability Scheduling partial schedule reserved capacity avoids later conflict constraint satisfaction + objective score Browser automation navigation/form state longer early route avoids blocked later state successful final transaction/state Data agent transformation pipeline more expensive cleanup prevents downstream validation failure schema/data validation Optimization partial candidate weak local score leads to superior global optimum objective function The same architecture appears in all of them.
What changes is the state, action space, rollout policy, and reward.
A Practical Decision Rule Use this progression.
Can one model call solve it reliably? ↓ no Can Best-of-N expose enough alternatives? ↓ no Can shallow scores reliably rank partial states? ↓ yes Use beam / Tree of Thoughts ↓ no Do branches have delayed consequences? ↓ yes Does repeated exploration improve estimates? ↓ yes Consider MCTS That is the right way to reach MCTS.
Not:
MCTS sounds advanced ↓ let's add MCTS If Your MCTS Agent Is Too Slow Search for the source of cost.
Too many expansions Reduce branching factor.
Use state-dependent action sets.
Use progressive widening.
Rollouts too expensive Use staged evaluation.
cheap heuristic ↓ only promising nodes ↓ expensive environment check Too much duplicate work Add state fingerprints and transposition reuse.
Too many similar rollouts Increase strategy diversity or reduce redundant sampling.
Tree keeps growing after answer is obvious Add evidence-based stopping conditions.
If MCTS Keeps Choosing the Wrong Branch Inspect four things.
1. Reward alignment Does the search reward actually correlate with verified success?
2. Exploration Are uncertain branches getting enough visits?
3. Rollout quality Are rollouts representative of what would actually happen?
4. Final selection Are you selecting by visits, mean value, verifier result, or some accidental mixture?
Do not immediately increase the iteration budget.
More search with the wrong reward often makes the wrong answer more confident.
If More Iterations Make Results Worse That is a very useful signal.
Possible causes include:
biased value model misaligned reward correlated rollouts exploration coefficient too low systematic simulator error final selector mismatch If search quality falls as compute rises, you have likely built an optimization process around the wrong signal.
That is more serious than simple randomness.
MCTS Is Not Intelligence It is easy to anthropomorphize the behavior.
The agent “considers alternatives.”
The agent “changes its mind.”
The agent “plans ahead.”
But operationally:
MCTS = structured allocation of finite search compute That is enough.
And it gives us a much better engineering question:
Does this allocation policy produce more verified successes per unit of compute than the simpler alternatives?
The Full Advanced Stack So Far We can now see the progression clearly.
chain of thought ↓ explicit intermediate computation self-consistency ↓ sample multiple completed reasoning trajectories Tree of Thoughts ↓ branch during reasoning and prune partial states MCTS ↓ adaptively allocate search compute using accumulated value evidence These are not four levels of intelligence.
They are four different compute-allocation mechanisms.
Use the smallest one that fixes the failure you can actually measure.
What Comes Next MCTS assumes the branches are competing trajectories through roughly the same problem-solving machinery.
But sometimes the deeper problem is not search.
It is specialization.
A cheap local model might be excellent at classification.
A code model might be best at patch generation.
A deterministic tool might be best at arithmetic.
A frontier model might only be needed for genuinely ambiguous reasoning.
Instead of asking:
Which branch should receive more search? we can ask:
Which expert should receive this task at all? That leads to the next post:
Advanced Agents From First Principles 05: Is One Model Doing Everything? Build a Mixture of Experts at the Agent Level.
And that takes us from search allocation to expert allocation.