A common failure in search-based agents is easy to miss.

The agent generates several plausible branches.

It scores them.

One branch looks weak.

So the runtime prunes it.

Later, you discover that the discarded branch was the only one that could have reached the correct solution.

The problem was not generation.

The problem was not necessarily the model.

The problem was search allocation.

The agent spent too much compute exploiting what looked good early and too little compute exploring alternatives whose value only became visible later.

That is the kind of problem Monte Carlo Tree Search is designed to address.

This article builds MCTS from first principles and asks the question that matters in production:

When does MCTS actually solve a problem that simpler search does not?

The answer is not “when the task is hard.”

The answer is closer to:

Use MCTS when early evaluations are unreliable, consequences unfold over multiple steps, and additional exploration can reveal delayed value that greedy pruning would otherwise destroy.

That distinction matters.

Because MCTS can also be expensive, noisy, slow, and completely unnecessary.

The Problem: Early Scores Can Lie

In the previous post we built Tree of Thoughts as search over intermediate semantic states.

The simplest version looked like this:

state
  ↓
branch
  ↓
evaluate
  ↓
keep top K
  ↓
branch again

That is often implemented as beam-style search.

At every depth, the best-looking partial states survive.

That works well when the evaluator is reasonably predictive of eventual success.

But suppose the search space looks like this:

root
├── A  score 0.90
│   └── A1 score 0.88
│       └── terminal FAIL
│
├── B  score 0.74
│   └── B1 score 0.80
│       └── terminal PASS
│
└── C  score 0.68
    └── C1 score 0.70
        └── terminal FAIL

A beam width of one chooses A.

A shallow heuristic says A is clearly best.

But the terminal evidence says B was the correct route.

The search algorithm has confused:

looks promising now

with:

produces good outcomes later

MCTS exists to manage that uncertainty.

Search Is Compute Allocation

It helps to stop thinking of MCTS as a mysterious game-playing algorithm.

At the agent level, MCTS is a way of deciding:

Which partial trajectory deserves the next unit of compute?

Suppose you have 100 model calls available.

You could spend them evenly:

A: 25
B: 25
C: 25
D: 25

Or greedily:

A: 70
B: 10
C: 10
D: 10

Or adaptively:

A initially looks strongest
↓
explore A
↓
A's deeper outcomes disappoint
↓
redirect compute toward B
↓
B begins producing stronger terminal evidence
↓
allocate more compute to B

That last pattern is the heart of MCTS.

It continuously balances two competing objectives:

exploit what currently looks good

and:

explore what remains uncertain

The Four Operations

A basic MCTS loop contains four operations:

1. selection
2. expansion
3. simulation / evaluation
4. backpropagation

Then it repeats.

root
 ↓
selection
 ↓
leaf
 ↓
expansion
 ↓
new child
 ↓
simulation / evaluation
 ↓
reward
 ↓
backpropagation
 ↓
update tree statistics
 ↓
repeat

Let’s build each part.

1. Selection

Selection chooses the existing tree node that should receive more search effort.

If we always choose the node with the highest average reward, we become greedy.

That creates a familiar failure:

first lucky branch
      ↓
looks best
      ↓
gets more samples
      ↓
other branches remain underexplored

We need a mechanism that rewards both:

• high estimated value
• insufficient exploration

A common form is UCB1.

For child node i:

UCB(i) = mean_reward(i)
       + c * sqrt(log(parent_visits) / child_visits(i))

The first term is exploitation.

mean_reward(i)

The second term is exploration.

sqrt(log(parent_visits) / child_visits(i))

The constant c controls how aggressively we explore.

Large c:

more exploration

Small c:

more exploitation

The important thing is not the exact equation.

The important thing is the policy:

A branch that looks good should receive more compute, but a branch that has barely been tested should not be discarded merely because its current estimate is uncertain.

A Tiny UCB Function

from __future__ import annotations

import math


def ucb_score(
    mean_reward: float,
    parent_visits: int,
    child_visits: int,
    exploration: float = 1.4,
) -> float:
    if child_visits == 0:
        return float("inf")

    return mean_reward + exploration * math.sqrt(
        math.log(max(parent_visits, 1)) / child_visits
    )

Unvisited children get infinite priority.

That forces the runtime to gather at least some evidence before assuming they are worthless.

2. Expansion

Once selection reaches a node that can be expanded, the agent generates one or more possible next states.

For a coding agent:

current repository state
├── inspect failing test
├── inspect implementation
├── inspect recent git diff
└── reproduce failure locally

For incident response:

current incident state
├── inspect application logs
├── inspect database latency
├── inspect queue depth
└── inspect deployment history

For research:

current hypothesis
├── search primary source A
├── search dataset B
├── test alternative explanation C
└── inspect contradiction D

Expansion is normally model-driven.

But it does not have to be.

The available actions might come from a deterministic action space.

That is often better when the domain already has clearly defined operations.

3. Simulation or Evaluation

Classic MCTS often performs a rollout from the newly expanded node until a terminal outcome is reached.

For an agent, that may be expensive.

A rollout could mean:

• multiple LLM calls
• tool execution
• running tests
• querying external systems
• simulated actions
• invoking a cheaper value model

So in practice we usually have several options.

Full rollout

partial state
 ↓
continue until terminal state
 ↓
measure terminal reward

Strong evidence.

Potentially expensive.

Partial rollout

partial state
 ↓
advance 2-3 steps
 ↓
evaluate resulting state

Cheaper.

Less reliable.

Value model

partial state
 ↓
learned scorer estimates eventual success

Very cheap.

But now MCTS depends heavily on value-model calibration.

Environment proxy

candidate patch
 ↓
run targeted tests
 ↓
reward from failing-test reduction

Often excellent when available.

For agent systems, this is crucial:

Use real environment evidence whenever the environment can cheaply reveal progress.

Do not replace a test suite with an LLM saying the patch “looks likely to work.”

4. Backpropagation

After evaluating a node, MCTS propagates the resulting reward back through the path that produced it.

Suppose the selected path was:

root
 ↓
B
 ↓
B3
 ↓
B3a

And the rollout reward was:

0.9

Then every node in the path updates its statistics.

For example:

node.visits += 1
node.value_sum += reward

Its mean value becomes:

mean = node.value_sum / node.visits

The important consequence is that early nodes slowly accumulate evidence about what their descendants tend to produce.

That is why MCTS can recover from misleading shallow evaluations.

A Minimal Node

from dataclasses import dataclass, field
from typing import Any


@dataclass
class SearchNode:
    state: Any
    parent: "SearchNode | None" = None
    action: str | None = None
    children: list["SearchNode"] = field(default_factory=list)
    visits: int = 0
    value_sum: float = 0.0
    terminal: bool = False

    @property
    def mean_value(self) -> float:
        if self.visits == 0:
            return 0.0
        return self.value_sum / self.visits

This node contains two different kinds of information.

The semantic state:

state

And the search statistics:

visits
value_sum
mean_value

That distinction becomes important later.

A Minimal MCTS Skeleton

class MCTS:
    def __init__(self, exploration: float = 1.4):
        self.exploration = exploration

    def select_child(self, node: SearchNode) -> SearchNode:
        return max(
            node.children,
            key=lambda child: ucb_score(
                mean_reward=child.mean_value,
                parent_visits=node.visits,
                child_visits=child.visits,
                exploration=self.exploration,
            ),
        )

    def backpropagate(self, node: SearchNode, reward: float) -> None:
        current = node

        while current is not None:
            current.visits += 1
            current.value_sum += reward
            current = current.parent

This is not yet a complete agent.

But the allocation mechanism is already visible.

Tree of Thoughts vs MCTS

This distinction gets confused constantly.

A typical beam-style Tree of Thoughts system asks:

Which K branches look best right now?

Then it keeps those branches.

MCTS asks something closer to:

Which branch should receive the next unit of search compute,
considering both what we know and what we still do not know?

That difference matters when value is delayed.

Beam-style search

evaluate frontier
 ↓
keep top K
 ↓
discard rest

Once a branch is pruned, it is gone.

MCTS

retain tree statistics
 ↓
allocate visits adaptively
 ↓
underexplored branches remain eligible

The branch does not have to look best immediately.

It has to justify further exploration relative to alternatives.

When MCTS Is Actually Useful

MCTS is especially attractive when four conditions hold.

1. Early scores are weak predictors

If a shallow score already predicts terminal success extremely well, beam search is simpler.

MCTS helps when:

partial-state score
        ↓
poorly predicts
        ↓
terminal outcome

2. The task has delayed consequences

For example:

architectural change
 ↓
seems larger initially
 ↓
removes entire class of downstream failures

A greedy search might prefer the smaller patch.

MCTS can keep allocating some compute to the more uncertain structural branch.

3. Branch statistics are reusable

Repeated visits must tell us something useful.

If every rollout is unrelated noise, backpropagating averages does little.

4. Search is expensive enough that allocation matters

If you can exhaustively evaluate every branch cheaply, you do not need MCTS.

Just evaluate everything.

When MCTS Is Overkill

There are many cases where MCTS should not be your first answer.

If the task is:

one prompt
 ↓
one good answer

use one model call.

If the task is:

generate 5 complete candidates
 ↓
verify each
 ↓
choose best

use Best-of-N.

If the task is:

branch a few times
 ↓
partial score is reliable
 ↓
keep top K

use beam search / Tree of Thoughts.

MCTS becomes interesting when:

branch quality is uncertain
+
reward arrives late
+
search budget is limited
+
repeated evidence improves value estimates

The Most Important Question: What Is the Reward?

A beautiful search policy cannot rescue a bad reward signal.

Suppose your coding agent rewards:

number of tests passed

That may encourage:

remove failing tests

or:

skip test execution

or:

modify assertions

depending on what actions the runtime permits.

The reward must represent the actual objective.

For example:

required tests pass
+
no regression tests fail
+
changed behavior matches specification
+
patch scope remains bounded

The exact weights matter less than the principle:

The reward function is part of the agent specification.

Reward Shaping Can Help—and Mislead

Terminal rewards are often sparse.

PASS = 1
FAIL = 0

That is clean but may provide little guidance.

We can add intermediate signals.

For a coding agent:

+0.2 target test failure reproduced
+0.3 number of failing tests reduced
+0.3 target test passes
+0.2 full suite passes

But reward shaping creates a new risk.

The agent may optimize the proxy instead of the goal.

So we should always retain a distinction between:

search reward

and:

final verification

The search reward guides compute.

The verifier decides whether the result is actually acceptable.

Search Reward Is Not Verification

This distinction is so important that it is worth making explicit.

high MCTS value
        ≠
verified success

A branch can accumulate excellent expected reward and still fail final acceptance.

Therefore the architecture should look like:

MCTS
 ↓
select candidate trajectory
 ↓
execute / materialize result
 ↓
external verifier
 ↓
PASS / FAIL / UNKNOWN

The tree decides where to search.

The verifier decides whether reality supports the result.

Coding Agent Example

Consider a coding agent debugging a failing integration test.

The root state contains:

repository state
failing test
error message
current diff

Initial branches might be:

A: patch failing function directly
B: inspect call graph
C: inspect recent schema change
D: inspect test fixture

A superficial score might rank A first.

It proposes the obvious fix.

But perhaps the real defect is a schema mismatch introduced two layers earlier.

MCTS can allocate some rollouts to B and C.

A rollout for B might become:

inspect call graph
 ↓
identify shared parser
 ↓
modify parser
 ↓
run targeted tests
 ↓
run regression suite

The terminal evidence can then flow backward to B.

Over repeated visits, the search policy learns that B produces stronger outcomes than A even though A looked better initially.

Coding Agents: A Good MCTS Reward

For code, the strongest reward often comes from the environment.

For example:

def coding_reward(result) -> float:
    score = 0.0

    if result.reproduced_failure:
        score += 0.10

    if result.target_test_passes:
        score += 0.40

    if result.regression_suite_passes:
        score += 0.40

    if result.diff_is_bounded:
        score += 0.10

    return score

But final acceptance still remains separate.

def verify_patch(result) -> bool:
    return (
        result.target_test_passes
        and result.regression_suite_passes
        and result.requirements_satisfied
    )

Research Agent Example

A research agent may have several competing explanations.

hypothesis A
hypothesis B
hypothesis C

Early evidence may favor A.

But additional evidence could overturn it.

MCTS can treat evidence acquisition itself as part of the action space.

A
├── inspect source supporting A
├── inspect source contradicting A
└── search for primary dataset

This is much healthier than repeatedly asking an LLM:

How confident are you in hypothesis A?

The search should spend compute reducing uncertainty through evidence.

Planning and Scheduling

Scheduling problems are another natural fit.

A partial schedule may look mediocre because it reserves capacity.

But that capacity may prevent a major conflict later.

Greedy planning often prefers immediate utilization.

MCTS can evaluate longer-horizon consequences.

Examples include:

• job-shop scheduling
• delivery routing
• resource allocation
• workflow orchestration
• project sequencing

The important condition remains the same:

Early local quality is not a reliable proxy for eventual global quality.

Incident Response

Incident-response agents also have delayed consequences.

Suppose the agent sees high database latency.

Possible actions:

restart database
inspect query plan
inspect recent deployment
inspect connection pool

Restarting might produce an immediate improvement.

That gives it a high short-term reward.

But if the root cause is a bad deployment, the problem returns.

A longer branch:

inspect deployment
 ↓
identify query regression
 ↓
rollback safely
 ↓
verify latency
 ↓
verify error rate

may have better long-term value.

This is exactly the kind of delayed-payoff structure where MCTS can become useful.

Browser and UI Agents

MCTS can also help browser agents when navigation choices create delayed constraints.

For example:

checkout
├── guest checkout
├── create account
└── login existing account

One path might appear shorter.

But later it may require information the agent does not have.

A more expensive early branch could reduce downstream uncertainty.

Again:

shortest immediate path
        ≠
best complete trajectory

The Action Space Matters

MCTS assumes the search tree contains useful actions.

If expansion generates nonsense, search cannot rescue it.

So we still need strong action-space design.

For every node ask:

What actions are actually legal here?

not:

What random ideas can the model imagine?

For a coding agent, legal actions might include:

read_file
search_symbol
run_test
inspect_diff
apply_patch

The runtime can constrain expansion according to state.

if no patch exists:
    do not offer revert_patch

This dramatically reduces branching noise.

Progressive Widening

Large agent action spaces create another problem.

Suppose each node can produce 50 plausible children.

Expanding them all immediately is expensive.

Progressive widening addresses this by allowing the number of children to grow with node visits.

Conceptually:

few visits
 ↓
few children

many visits
 ↓
allow more children

That lets the search deepen promising nodes before exploding the branching factor.

A simple policy might be:

allowed_children = max(1, int(node.visits ** 0.5))

The exact schedule should be benchmarked.

The principle is what matters.

Value Noise

LLM-based value functions can be extremely noisy.

Ask the same model to score the same partial branch repeatedly and you may get:

0.82
0.67
0.91
0.73

If MCTS treats each score as precise, the search statistics become misleading.

Useful diagnostics include:

value mean
value variance
rank stability
confidence interval

For high-noise evaluators, repeated evaluation may be justified.

But before adding more model calls, ask whether the environment offers a stronger signal.

Correlated Rollouts

Ten rollouts are not necessarily ten independent pieces of evidence.

If every rollout uses:

• the same model
• the same prompt
• the same retrieval context
• the same tool policy

then their errors may be highly correlated.

You can have:

10 visits

but much less than:

10 independent observations

This is the same problem we saw with self-consistency.

So rollout diversity matters.

Possible variations include:

• strategy-conditioned rollout policies
• different model tiers
• deterministic environment checks
• different retrieved evidence
• different tool routes

But diversity should be purposeful.

Randomness alone is not evidence quality.

Exploration Constant Failure Modes

The exploration coefficient controls how aggressively MCTS revisits uncertain branches.

Too low:

search becomes nearly greedy

Too high:

search wastes compute on obviously weak branches

Do not treat 1.4 as a magical constant just because it appears in examples.

Sweep it.

c = 0.25
c = 0.5
c = 1.0
c = 1.4
c = 2.0

Then measure verified outcomes, not tree aesthetics.

Search Budgets Must Be Explicit

MCTS is dangerous without budgets because the algorithm always has another branch worth investigating.

Bound at least:

max iterations
max tree nodes
max model calls
max tool calls
max wall time
max cost
max depth

For example:

@dataclass
class SearchBudget:
    max_iterations: int = 100
    max_nodes: int = 500
    max_model_calls: int = 250
    max_seconds: float = 60.0
    max_cost_usd: float = 2.0

A production search system should have a named termination reason.

VERIFIED_SUCCESS
ITERATION_BUDGET
NODE_BUDGET
TIME_BUDGET
COST_BUDGET
NO_PROGRESS
NO_VALID_ACTIONS

Early Stopping

MCTS does not need to consume the entire budget.

Useful stopping conditions include:

verified terminal success

or:

best branch value stable for N iterations

or:

additional search produces no meaningful improvement

or:

value gap between best and alternatives exceeds threshold

But be careful.

Stopping because one branch’s model score looks dominant can recreate the exact premature-pruning problem MCTS was introduced to solve.

Whenever possible, use stronger evidence.

Do Not Search Irreversible Side Effects

This rule from the earlier search post becomes even more important with MCTS.

You do not want rollouts that repeatedly:

send emails
charge credit cards
restart production services
merge pull requests
delete data

Search should usually operate over:

• plans
• simulated state
• sandboxed environments
• temporary worktrees
• read-only diagnostics
• reversible transformations

Then select a trajectory.

Then execute the real side effect once.

search
 ↓
select
 ↓
verify preconditions
 ↓
commit irreversible action
 ↓
verify postconditions

A More Complete MCTS Skeleton

Here is a deliberately small implementation structure.

from dataclasses import dataclass
from typing import Protocol


class Environment(Protocol):
    def actions(self, state): ...
    def transition(self, state, action): ...
    def is_terminal(self, state) -> bool: ...
    def reward(self, state) -> float: ...


@dataclass
class MCTSConfig:
    iterations: int = 100
    exploration: float = 1.4
    max_depth: int = 8


class MCTSAgent:
    def __init__(self, env: Environment, config: MCTSConfig):
        self.env = env
        self.config = config

    def search(self, root: SearchNode) -> SearchNode:
        for _ in range(self.config.iterations):
            leaf = self._select(root)
            child = self._expand(leaf)
            reward = self._evaluate(child)
            self._backpropagate(child, reward)

        return self._best_root_child(root)

Now the methods.

    def _select(self, node: SearchNode) -> SearchNode:
        current = node

        while current.children and not current.terminal:
            current = max(
                current.children,
                key=lambda child: ucb_score(
                    child.mean_value,
                    current.visits,
                    child.visits,
                    self.config.exploration,
                ),
            )

        return current

Expansion:

    def _expand(self, node: SearchNode) -> SearchNode:
        if node.terminal:
            return node

        actions = self.env.actions(node.state)

        if not actions:
            node.terminal = True
            return node

        action = actions[0]
        next_state = self.env.transition(node.state, action)

        child = SearchNode(
            state=next_state,
            parent=node,
            action=str(action),
            terminal=self.env.is_terminal(next_state),
        )

        node.children.append(child)
        return child

Evaluation:

    def _evaluate(self, node: SearchNode) -> float:
        return self.env.reward(node.state)

Backpropagation:

    def _backpropagate(self, node: SearchNode, reward: float) -> None:
        current = node

        while current is not None:
            current.visits += 1
            current.value_sum += reward
            current = current.parent

And final selection:

    def _best_root_child(self, root: SearchNode) -> SearchNode:
        return max(root.children, key=lambda child: child.visits)

This is intentionally incomplete.

A serious implementation still needs:

• unexpanded-action tracking
• rollout policies
• duplicate-state handling
• budgets
• provenance
• concurrency control
• state hashing
• failure handling
• verification

But the architecture is visible.

Why Select by Visits at the End?

During search we need exploration.

At the end, we do not.

So the final decision often uses:

most visited child

or:

highest mean value child

rather than UCB.

The exploration bonus is a search allocation device.

It should not automatically become the production decision rule.

Transpositions and Duplicate States

Agent trees often contain different paths that reach the same state.

For example:

read tests → inspect implementation

and:

inspect implementation → read tests

may lead to the same knowledge state.

If we represent them as completely independent nodes, the tree duplicates work.

A state fingerprint can help.

import hashlib
import json


def state_fingerprint(state: dict) -> str:
    canonical = json.dumps(state, sort_keys=True, default=str)
    return hashlib.sha256(canonical.encode()).hexdigest()

Then the runtime can detect repeated states.

This converts the search structure from a pure tree toward a graph.

That can substantially improve efficiency.

Provenance Is Mandatory

Advanced search quickly becomes impossible to debug without lineage.

Every node should record at least:

node_id
parent_id
action
state fingerprint
visit count
value mean
value variance
creation source
model / tool used
observations
verification evidence

Then when a surprising branch wins, you can reconstruct why.

Without provenance, MCTS becomes a stochastic black box wrapped around another stochastic black box.

That is not a production architecture.

Parallel MCTS

Agent workloads often involve expensive model or tool calls.

Parallel search is tempting.

But naive parallelism creates a problem.

Several workers may all select the same currently promising node before any of them updates its statistics.

That duplicates effort.

A common technique is a virtual loss.

Temporarily penalize a node while a worker explores it.

worker selects node
 ↓
apply virtual loss
 ↓
other workers prefer alternatives
 ↓
worker finishes rollout
 ↓
remove virtual loss
 ↓
backpropagate real reward

This is useful when model calls dominate latency.

But parallelism should still be benchmarked against cost and duplicated work.

Learned Value Models

The model series now becomes directly useful again.

Instead of asking an LLM judge to score every node, we could use a learned value model.

Conceptually:

partial agent state
 ↓
encoder
 ↓
value model
 ↓
estimated probability of eventual success

That value model might be simple.

For example:

MR.Q-style scalar scorer

or recurrent if trajectory history matters.

But the evidence rule remains:

A value model earns its place only if it predicts eventual verified outcomes well enough to improve search allocation.

Do not use it because the architecture diagram looks sophisticated.

Calibrating the Value Model

Suppose the scorer predicts:

0.8

for 100 states.

Roughly 80 should eventually succeed if the score is well calibrated as a probability.

Useful diagnostics include:

Brier score
calibration curve
rank correlation
AUC
precision among top-ranked branches

But the most important metric remains downstream:

Does MCTS using this scorer improve verified task success?

Failure Attribution

When MCTS fails, do not say:

MCTS did not work

Localize the failure.

No correct branch ever appears

generator / action-space failure

Correct branch appears but receives few visits

selection / exploration failure

Correct branch receives visits but low rewards

evaluator / rollout failure

Correct branch has good evidence but loses at root selection

aggregation / final-selection failure

Correct branch wins search but task still fails

execution / verification failure

That decomposition is extremely useful in production.

Search Metrics

For every run, log:

iterations
nodes created
nodes revisited
max depth
branching factor
rollouts
model calls
tool calls
value evaluations
unique states
wall time
cost
termination reason
verified outcome

Then add search-quality metrics.

oracle generated success
best verified branch discovered
visit concentration
value variance
root action entropy
pruning / allocation regret

For example:

oracle discovered success = true
selected branch verified success = false

This immediately tells us the search generated a winning trajectory but failed to allocate/select correctly.

The Experiment That Justifies MCTS

Do not compare MCTS against nothing.

Compare it against increasingly complex baselines.

A: one-shot agent
B: Best-of-N
C: Tree of Thoughts / beam width 2
D: Tree of Thoughts / beam width 4
E: MCTS 25 iterations
F: MCTS 100 iterations

Measure:

verified success
oracle discovered success
model calls
tool calls
latency
cost
cost per verified success

Then ask:

Does adaptive exploration outperform simple frontier pruning?

If not, keep the simpler system.

Example Benchmark Table

architecture      success   calls   latency   cost
---------------------------------------------------
one-shot            62%        1      1.0x    1.0x
Best-of-5           71%        5      1.8x    5.0x
beam width 3        78%       14      3.2x   14.0x
MCTS 25             79%       23      4.5x   23.0x
MCTS 100            80%       88     12.0x   88.0x

If your real data looks like this, MCTS may not be worth it.

The gain from beam search to MCTS is tiny.

A better result might look like:

architecture      success   calls
----------------------------------
beam width 3        61%       14
MCTS 25             74%       23

Now the adaptive search policy may be solving a real delayed-value problem.

Application Matrix

The same architecture appears in all of them.

What changes is the state, action space, rollout policy, and reward.

A Practical Decision Rule

Use this progression.

Can one model call solve it reliably?
        ↓ no
Can Best-of-N expose enough alternatives?
        ↓ no
Can shallow scores reliably rank partial states?
        ↓ yes
Use beam / Tree of Thoughts
        ↓ no
Do branches have delayed consequences?
        ↓ yes
Does repeated exploration improve estimates?
        ↓ yes
Consider MCTS

That is the right way to reach MCTS.

Not:

MCTS sounds advanced
        ↓
let's add MCTS

If Your MCTS Agent Is Too Slow

Search for the source of cost.

Too many expansions

Reduce branching factor.

Use state-dependent action sets.

Use progressive widening.

Rollouts too expensive

Use staged evaluation.

cheap heuristic
 ↓
only promising nodes
 ↓
expensive environment check

Too much duplicate work

Add state fingerprints and transposition reuse.

Too many similar rollouts

Increase strategy diversity or reduce redundant sampling.

Tree keeps growing after answer is obvious

Add evidence-based stopping conditions.

If MCTS Keeps Choosing the Wrong Branch

Inspect four things.

1. Reward alignment

Does the search reward actually correlate with verified success?

2. Exploration

Are uncertain branches getting enough visits?

3. Rollout quality

Are rollouts representative of what would actually happen?

4. Final selection

Are you selecting by visits, mean value, verifier result, or some accidental mixture?

Do not immediately increase the iteration budget.

More search with the wrong reward often makes the wrong answer more confident.

If More Iterations Make Results Worse

That is a very useful signal.

Possible causes include:

biased value model
misaligned reward
correlated rollouts
exploration coefficient too low
systematic simulator error
final selector mismatch

If search quality falls as compute rises, you have likely built an optimization process around the wrong signal.

That is more serious than simple randomness.

MCTS Is Not Intelligence

It is easy to anthropomorphize the behavior.

The agent “considers alternatives.”

The agent “changes its mind.”

The agent “plans ahead.”

But operationally:

MCTS = structured allocation of finite search compute

That is enough.

And it gives us a much better engineering question:

Does this allocation policy produce more verified successes per unit of compute than the simpler alternatives?

The Full Advanced Stack So Far

We can now see the progression clearly.

chain of thought
    ↓
explicit intermediate computation

self-consistency
    ↓
sample multiple completed reasoning trajectories

Tree of Thoughts
    ↓
branch during reasoning and prune partial states

MCTS
    ↓
adaptively allocate search compute using accumulated value evidence

These are not four levels of intelligence.

They are four different compute-allocation mechanisms.

Use the smallest one that fixes the failure you can actually measure.

What Comes Next

MCTS assumes the branches are competing trajectories through roughly the same problem-solving machinery.

But sometimes the deeper problem is not search.

It is specialization.

A cheap local model might be excellent at classification.

A code model might be best at patch generation.

A deterministic tool might be best at arithmetic.

A frontier model might only be needed for genuinely ambiguous reasoning.

Instead of asking:

Which branch should receive more search?

we can ask:

Which expert should receive this task at all?

That leads to the next post:

Advanced Agents From First Principles 05: Is One Model Doing Everything? Build a Mixture of Experts at the Agent Level.

And that takes us from search allocation to expert allocation.