# Advanced Agents From First Principles 03: Does Your Agent Commit to a Bad Reasoning Path Too Early? Build a Tree of Thoughts
# Book: Advanced Agents From First Principles
# Chapter URL: https://programmer.ie/books/advanced-agents-from-first-principles/03-chapter/
# Prose fingerprint (sha256, normalized): b2e1eafc0e59c61e92899d32fe08575d590e78ca0947e53065175e11f0f9b7b4
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

A reasoning agent can fail even when every individual step looks plausible.
The problem is often not that the model cannot produce a good line of reasoning.
The problem is that it commits too early.
It chooses one interpretation, one hypothesis, one plan, or one next step and then spends the rest of the run trying to make that decision work.
That gives us a common failure pattern:
problem ↓ first plausible thought ↓ second thought conditioned on the first ↓ third thought conditioned on both ↓ ... ↓ confident answer built on an early mistake If the first branch was wrong, every later step inherits the error.
A Tree of Thoughts changes the shape of the computation.
Instead of asking:
What should I think next?
we ask:
What are several plausible next reasoning states, and which ones deserve more computation?
The runtime becomes responsible for preserving alternatives long enough for evidence or evaluation to discriminate between them.
The resulting structure looks like this:
root / | \ A B C / | \ / \ / \ A1 A2 A3 B1 B2 C1 C2 ↓ ↓ ↓ score / verify / prune ↓ ↓ ↓ keep frontier ↓ expand again This is not simply “more chain of thought.”
It is search over intermediate reasoning states.
And that distinction matters.
The searchable problem If you found this article because you searched for something like:
AI agent commits to wrong reasoning path LLM reasoning gets stuck on first idea Tree of Thoughts Python agent reasoning search how to make AI explore multiple solutions LLM keeps following bad assumptions reasoning agent cannot backtrack AI agent search tree Tree of Thoughts vs chain of thought Tree of Thoughts vs beam search Tree of Thoughts vs MCTS then the first thing to understand is this:
Do not add Tree of Thoughts because the task is difficult. Add it because early commitment is a measured failure mode.
A difficult task does not automatically require branching.
If a single reasoning trajectory already succeeds reliably, branching only adds cost and latency.
Tree of Thoughts becomes useful when:
there are several plausible intermediate interpretations or strategies, early choices materially affect later success, weak branches can be identified before full completion, the cost of exploring alternatives is lower than the cost of repeatedly repairing a bad commitment. 1. Chain of thought is a path In the previous post, we treated reasoning as intermediate computation.
A simplified reasoning state might look like:
from dataclasses import dataclass, field @dataclass class ReasoningState: goal: str assumptions: list[str] = field(default_factory=list) hypotheses: list[str] = field(default_factory=list) evidence: list[str] = field(default_factory=list) unknowns: list[str] = field(default_factory=list) A normal reasoning loop evolves one state:
S0 → S1 → S2 → S3 → result That is a path.
The agent can revise the path later, but its default trajectory is still linear.
The weakness is obvious when several plausible choices exist at S1.
A linear system picks one.
A tree system preserves several.
2. Tree of Thoughts turns reasoning into search Instead of:
next_state = reason(current_state) we ask for multiple candidate successors:
next_states = propose_next_states(current_state, n=4) Now the runtime has a choice.
It can inspect the candidates, gather evidence, score them, reject duplicates, and expand only the most promising ones.
The basic algorithm is:
frontier = [root] repeat: expand each frontier node evaluate children prune weak / duplicate children keep best frontier stop if solved or budget exhausted This is the core mechanism.
Everything else is policy.
3. A thought should be a state transition, not a paragraph One of the easiest mistakes is to treat a “thought” as arbitrary prose.
That makes search difficult because the runtime has no stable structure to evaluate.
A better representation is explicit.
For example:
from dataclasses import dataclass, field from typing import Literal @dataclass class Thought: strategy: str claim: str assumptions: list[str] evidence_needed: list[str] expected_value: float | None = None status: Literal["candidate", "rejected", "verified"] = "candidate" Now a branch means something concrete.
For a coding agent, one thought might be:
strategy: inspect failing test first claim: regression is in parser normalization assumptions: - failure started after parser refactor - test fixture is valid evidence_needed: - git diff for parser - failing test output Another branch might be:
strategy: inspect dependency boundary claim: runtime passes malformed input before parser assumptions: - parser itself may be correct evidence_needed: - call-site trace - input sample These are not merely two phrasings of the same idea.
They are different hypotheses with different evidence requirements.
That is useful branching.
4. The most important question: what makes branches meaningfully different? Sampling four outputs does not guarantee four distinct reasoning paths.
You can easily get:
A: inspect the parser B: inspect parser behavior C: look at the parser implementation D: check parser code That is one branch written four ways.
A useful tree needs branch diversity.
You can encourage it by conditioning generation on distinct strategy families.
For example:
STRATEGIES = [ "test-first", "data-flow", "dependency-boundary", "historical-diff", "invariant-check", ] Then ask the model to produce one next thought under each strategy.
That changes the search space from:
same prompt × different random seed to:
same problem × different reasoning strategy That is much more valuable.
5. Deduplicate branches before spending more compute Tree search becomes expensive very quickly.
If every node produces four children:
depth 0: 1 depth 1: 4 depth 2: 16 depth 3: 64 depth 4: 256 Most practical systems therefore prune aggressively.
The first pruning step should often be deduplication.
A simple lexical fingerprint can remove obvious duplicates:
import hashlib import re def fingerprint(text: str) -> str: normalized = re.sub(r"\s+", " ", text.lower()).strip() return hashlib.sha256(normalized.encode()).hexdigest() Semantic deduplication can go further using embeddings.
But the principle is the same:
Do not pay to expand the same idea twice.
Useful metrics include:
unique_branch_ratio = unique_branches / generated_branches If your unique-branch ratio is 0.25, your “tree” may actually be four copies of the same path.
6. Evaluation is the hard part Generating branches is easy.
Choosing which branches deserve more compute is the difficult part.
Suppose the frontier contains:
A: likely parser regression B: likely database transaction bug C: likely stale cache interaction D: likely test fixture error How should the runtime choose?
There are several possible evaluators.
6.1 Model self-score The cheapest conceptual option is:
score = model_evaluate(thought) This can be useful, but it has obvious weaknesses.
The same model that generated a branch may prefer branches that resemble its own initial assumptions.
That creates correlated generation and evaluation errors.
Use model scores as a signal, not as truth.
6.2 Heuristic score Some domains provide cheap deterministic signals.
A coding agent can ask:
Does this hypothesis explain the failing test? Does the referenced file appear in the stack trace? Did the suspected module change recently? A research agent can ask:
Does this hypothesis have primary-source support? How many claims remain unsupported? A planning agent can ask:
Does this branch violate a hard constraint? Is the estimated cost within budget? These signals are often stronger than model preference.
6.3 Learned scorer The models from the earlier series fit naturally here.
A learned scorer can expose a stable interface:
def score(state, thought) -> float: ... The implementation might be a small ranker or task-specific model.
The important point is architectural:
The tree search does not need to know how the score is produced.
It only needs a comparable signal.
6.4 Environment evidence This is usually the strongest option when available.
For a coding agent:
run targeted test inspect stack trace check type checker measure benchmark For a browser agent:
inspect DOM state check URL verify form field value For a data agent:
run validation query check row counts compare schema A useful advanced-agent principle is:
Use the model to propose branches. Use the environment to eliminate them.
7. A minimal Tree of Thoughts runtime Here is a small standalone skeleton.
from __future__ import annotations from dataclasses import dataclass, field from typing import Callable @dataclass class Node: id: str state: str parent_id: str | None depth: int score: float = 0.0 terminal: bool = False metadata: dict = field(default_factory=dict) class TreeOfThoughts: def __init__( self, expand: Callable[[Node], list[Node]], evaluate: Callable[[Node], float], verify: Callable[[Node], bool], beam_width: int = 4, max_depth: int = 6, max_nodes: int = 100, ): self.expand = expand self.evaluate = evaluate self.verify = verify self.beam_width = beam_width self.max_depth = max_depth self.max_nodes = max_nodes self.generated = 0 def run(self, root: Node) -> Node | None: frontier = [root] for _ in range(self.max_depth): children: list[Node] = [] for node in frontier: if self.verify(node): return node new_nodes = self.expand(node) self.generated += len(new_nodes) for child in new_nodes: child.score = self.evaluate(child) children.append(child) if self.generated >= self.max_nodes: break if not children: return None children = deduplicate(children) children.sort(key=lambda n: n.score, reverse=True) frontier = children[: self.beam_width] if self.generated >= self.max_nodes: break for node in frontier: if self.verify(node): return node return max(frontier, key=lambda n: n.score, default=None) This is intentionally simple.
It gives us the essential mechanism:
expand ↓ evaluate ↓ deduplicate ↓ prune ↓ continue 8. Tree of Thoughts vs normal chain of thought A linear reasoning trace is:
A → B → C → D Tree of Thoughts is:
A / | \ B C D | / \ | E F G H The difference is not verbosity.
The difference is preserved alternatives.
Use chain of thought when:
the next step is usually clear, intermediate reasoning mainly decomposes a task, branching adds little value. Use Tree of Thoughts when:
several next steps are plausible, early decisions are consequential, weak branches can be rejected before completion. 9. Tree of Thoughts vs self-consistency Self-consistency usually samples complete trajectories:
root → A → B → answer 1 root → C → D → answer 2 root → E → F → answer 3 Then it aggregates the final outcomes.
Tree of Thoughts branches during the reasoning process:
root / | \ A B C / \ \ A1 A2 C1 That lets the runtime allocate more computation to promising partial trajectories.
This matters when completing every trajectory would be expensive.
10. Tree of Thoughts vs Best-of-N Best-of-N is usually:
prompt ↓ N complete candidates ↓ rank Tree of Thoughts is:
partial state ↓ N partial continuations ↓ prune ↓ expand survivors Best-of-N is often simpler and cheaper.
Use it first when complete candidate generation is affordable.
Tree search becomes more attractive when:
solutions are long, partial failures are detectable, full trajectories are expensive, branch quality diverges early. 11. Tree of Thoughts vs beam search This distinction is partly about representation and partly about evaluation.
A beam search typically keeps the top k partial sequences according to a score.
A Tree of Thoughts system applies the same general search idea to semantic reasoning states.
In practice, many Tree of Thoughts implementations are beam-search-like.
That is fine.
Do not invent a mystical boundary where one does not exist.
A practical implementation may simply be:
semantic state representation + branch generator + beam search + external verification The value comes from the architecture, not the label.
12. Tree of Thoughts vs MCTS This is an important boundary for this series.
Tree of Thoughts says:
Preserve and evaluate multiple reasoning branches.
MCTS adds a more sophisticated policy for deciding which branch to explore next.
MCTS becomes useful when:
early scores are weak predictors of final success, delayed payoff matters, exploration of uncertain branches is valuable, branch statistics improve with repeated visits, rollout outcomes provide better evidence than shallow heuristics. We will cover that later.
For now, Tree of Thoughts is the simpler foundation.
13. The branch-collapse problem A common failure looks like this:
root ├─ investigate parser ├─ inspect parser ├─ debug parser └─ review parser The system appears to branch but actually does not.
Useful branch-diversity diagnostics include:
unique strategy count unique evidence request count semantic similarity between siblings action-path diversity hypothesis diversity You can measure sibling similarity directly.
If all siblings are nearly identical, force strategy diversification before expanding further.
14. The evaluator-collapse problem Another failure happens when the evaluator assigns nearly identical scores to every branch:
A = 0.82 B = 0.81 C = 0.82 D = 0.81 The search policy now has almost no information.
Track:
score variance rank stability pairwise ordering consistency margin between retained and pruned branches If branch scores are effectively noise, more search may make the system worse.
The right response may be:
improve evaluator acquire stronger evidence reduce search depth switch to deterministic tests not:
increase tree width 15. Pruning regret Pruning creates a dangerous failure mode.
The correct branch may be generated and then discarded.
That is pruning regret.
A useful offline diagnostic is:
oracle_generated_success Ask:
Did any generated branch eventually contain a successful solution?
Then compare it with:
selected_success If oracle_generated_success is high but selected success is low, your generator may be fine.
Your search policy is the problem.
This is the same general lesson we saw in Best-of-N:
Separate generation failure from selection failure.
16. Search budgets must be explicit Without budgets, branching systems expand until cost explodes.
Useful limits include:
max_depth = 6 beam_width = 4 max_nodes = 100 max_model_calls = 30 max_tool_calls = 20 max_seconds = 60 max_cost = 0.50 A production runtime should know why search stopped.
For example:
class StopReason: VERIFIED = "verified" MAX_DEPTH = "max_depth" MAX_NODES = "max_nodes" MAX_COST = "max_cost" NO_PROGRESS = "no_progress" NO_FRONTIER = "no_frontier" Never let “the model stopped generating” become your termination policy.
17. Adaptive width is often better than fixed width Not every depth deserves the same branching factor.
Early exploration may benefit from width:
depth 0: width 6 depth 1: width 4 depth 2: width 3 depth 3: width 2 This creates a useful pattern:
wide early ↓ collect evidence ↓ narrow later Another strategy is uncertainty-based width.
If the frontier scores are close:
0.72 0.71 0.70 0.69 preserve more branches.
If one branch dominates:
0.92 0.55 0.42 0.31 narrow aggressively.
This is another form of adaptive compute allocation.
18. Search should prefer information before commitment Suppose a coding agent is considering two branches:
A: rewrite parser B: inspect input boundary Before editing anything, it may be cheaper to run a diagnostic test.
That gives us a useful search principle:
Prefer low-cost actions that reduce uncertainty before high-cost actions that commit the environment.
This applies across domains.
Coding:
read diff before editing run targeted test before refactor Research:
open primary source before writing conclusion Browser automation:
inspect current page state before clicking irreversible action Data systems:
validate sample before transforming full dataset 19. Application: coding agents Tree of Thoughts is especially useful when several engineering hypotheses are plausible.
Example problem:
CI began failing after a large refactor. Possible root branches:
A: dependency regression B: API mismatch C: fixture drift D: concurrency bug E: packaging/import issue Each branch can request cheap evidence:
stack trace changed files dependency diff test history runtime logs Then the runtime prunes weak hypotheses before making code changes.
A useful coding architecture is:
failure ↓ branch hypotheses ↓ collect cheap evidence ↓ prune ↓ create isolated patch branches/worktrees ↓ run tests ↓ select verified patch This is much safer than editing production state during speculative search.
20. Application: research agents Research questions frequently admit several interpretations.
Example:
Why did metric X decline after policy Y? Possible branches:
A: causal effect of policy B: measurement change C: economic confounder D: seasonal effect E: reporting lag A useful search tree asks different evidence questions for each branch.
The evaluator can prioritize:
primary-source support independent evidence contradiction count missing evidence causal plausibility The tree is valuable because the system can preserve multiple explanations until evidence eliminates them.
21. Application: debugging and incident response Incident-response agents are a natural fit because premature commitment is dangerous.
Suppose latency spikes suddenly.
Branches might include:
A: database saturation B: network degradation C: cache miss storm D: dependency slowdown E: deployment regression Cheap information-producing actions include:
read metrics inspect deploy timeline query traces check error rates compare regions The runtime can prune hypotheses before executing remediation.
This preserves an important safety boundary:
search over diagnosis freely commit remediation cautiously 22. Application: planning agents A planning problem may have several possible partial plans.
Example:
Schedule five tasks with resource and dependency constraints. Branches may represent different early allocations.
The evaluator can use deterministic constraint checks to reject impossible partial plans immediately.
That makes Tree of Thoughts valuable because partial invalidity is cheap to detect.
23. Application: analytics agents Suppose an analytics agent sees a revenue decline.
It might branch into:
A: acquisition decline B: conversion decline C: retention decline D: pricing mix shift E: reporting artifact Each branch triggers different queries.
The tree therefore becomes a mechanism for structured investigative analysis, not merely text generation.
24. Application: design-space exploration Some problems do not have one obviously correct solution.
Architecture design is an example.
Possible branches might be:
A: monolith B: modular monolith C: event-driven services D: workflow engine The runtime can score branches against constraints:
latency operational complexity team size failure isolation cost migration difficulty This is not proof that one architecture is objectively best.
It is structured exploration of a constrained design space.
25. Application matrix Software type Useful branches Cheap evidence Expensive commitment Coding agent root-cause hypotheses tests, diffs, traces code edits, PRs Research agent competing explanations primary sources published conclusion Incident agent failure hypotheses metrics, logs, traces restart, rollback, failover Planning agent partial schedules constraint checks reservation / allocation Analytics agent causal explanations queries, segments business recommendation Browser agent navigation strategies DOM/state inspection submit / purchase / delete Architecture agent design alternatives constraint scoring implementation commitment The same mechanism appears across all of them:
preserve alternatives ↓ acquire discriminating evidence ↓ prune ↓ commit later 26. Search is not automatically better reasoning A common mistake is:
more branches = smarter agent No.
More branches can mean:
more duplicate ideas more evaluator noise more latency more cost more opportunities to prune the correct path The correct question is:
Does branching increase verified success enough to justify its cost?
27. Benchmark the mechanism A useful experiment compares:
A: direct generation B: chain of thought C: self-consistency D: Best-of-N E: Tree of Thoughts width 2 F: Tree of Thoughts width 4 G: Tree of Thoughts width 8 Track:
verified success oracle generated success pruning regret unique branch ratio nodes generated nodes expanded model calls tool calls latency cost per verified success Do not evaluate only final answer quality.
You need to understand where the tree helped or failed.
28. A useful experiment record You can make architecture claims falsifiable.
from dataclasses import dataclass @dataclass class SearchExperiment: task_set: str baseline: str variant: str beam_width: int max_depth: int verified_success_rate: float oracle_success_rate: float pruning_regret: float avg_model_calls: float avg_latency_ms: float avg_cost: float Then the decision becomes empirical.
For example:
Tree of Thoughts width 4 +6.2 percentage points verified success +2.8× model calls +1.9× latency Now you can decide whether the trade-off is justified for the application.
29. When Tree of Thoughts is probably the wrong tool Do not use it when:
the next action is deterministic, objective verification already guides a simple loop, complete candidate generation is cheap enough for Best-of-N, branches cannot be evaluated until the very end, evaluator quality is extremely poor, latency dominates the product requirement, side effects cannot be isolated safely. In those cases, a simpler architecture is likely better.
30. The escalation ladder A useful progression is:
single trajectory ↓ reasoning state ↓ self-consistency ↓ Best-of-N ↓ Tree of Thoughts / beam-style search ↓ MCTS Each step adds a new mechanism.
Each step must solve a failure the previous one could not solve cheaply enough.
31. What Tree of Thoughts teaches us about agents The deepest lesson is not “use trees.”
It is this:
Reasoning quality can depend on how the runtime allocates computation across competing partial hypotheses.
The model generates possibilities.
The runtime decides:
which alternatives survive which evidence to gather which branches receive more compute when uncertainty is low enough to commit That is where advanced agent architecture begins to become distinct from prompting.
32. A production checklist Before shipping a Tree of Thoughts agent, ask:
[ ] Is premature commitment a measured failure? [ ] Are branches semantically distinct? [ ] Can weak branches be rejected early? [ ] Is the evaluator better than random preference? [ ] Do we have external evidence where possible? [ ] Are duplicate branches removed? [ ] Are width/depth/node/cost budgets explicit? [ ] Are irreversible side effects isolated from speculative search? [ ] Can we measure pruning regret? [ ] Does Tree of Thoughts beat simpler baselines? If several of those answers are no, the tree is probably architectural theater.
33. The key debugging map If the agent still fails after adding Tree of Thoughts, diagnose the failure by stage.
No good branch generated ↓ branch generator problem Good branch generated but pruned ↓ evaluator / pruning problem Good branch survives but never expanded ↓ search allocation problem Correct branch selected but execution fails ↓ executor problem Correct execution but task still fails ↓ verification / goal-definition problem This decomposition is much more useful than saying:
The agent reasoning was bad.
34. Final rule Tree of Thoughts is not valuable because it resembles human deliberation.
It is valuable when it gives the runtime a measurable ability to avoid premature commitment.
The governing rule is:
Branch when multiple partial hypotheses are genuinely plausible. Prune with evidence. Commit only when one branch has earned more confidence than the alternatives.
And remember the boundary that runs through this entire series:
plausible branch ≠ verified branch ≠ verified outcome Search changes how computation is allocated.
Verification still decides whether the result worked.
Next: Monte Carlo Tree Search Tree of Thoughts still relies heavily on shallow branch scores and beam-style pruning.
That can fail when the best branch looks weak early but produces strong outcomes later.
The next post will tackle exactly that problem:
Advanced Agents From First Principles 04: Does Your Agent Prune Good Ideas Too Early? Use Monte Carlo Tree Search for Long-Horizon Reasoning.