Is Your Advanced Agent Actually Better? Benchmark It Under Equal Budgets

You replace one model call with eight.

Success rises from 62% to 74%.

Great.

Except the new system used:

• eight times the inference,
• three extra judges,
• two rounds of critique,
• a larger context,
• a stronger verifier,
• and several times the latency.

Did the architecture improve?

Or did you just buy more attempts?

This is one of the easiest mistakes to make in advanced agent engineering.

A complicated system almost always has more opportunities to stumble into a correct answer than a one-shot baseline.

That does not prove the complicated system is better.

The central rule of this post is:

Compare architectures under equal resource budgets and external verification.

If MCTS gets ten model calls, compare it with alternatives that also get ten model calls.

If a mixture-of-agents system uses a frontier model for difficult tasks, charge that usage to the system.

If debate invokes three agents and a judge, count all four calls.

If a search method expands 100 nodes but only returns one answer, all 100 nodes belong in the cost.

Advanced agents are systems.

Benchmark the whole system.

The benchmark target is not “looks better”

The most useful top-level metric is usually something like:

verified successful tasks
-------------------------
total tasks

But even that is not enough.

You also care about what the success cost.

A practical benchmark should normally track at least:

verified success rate
cost per verified success
latency
model calls
input tokens
output tokens
tool calls
false-success rate
UNKNOWN rate

The important word is verified.

An agent saying “done” is not a successful task.

A critic saying “this looks correct” is not a successful task.

A majority vote is not a successful task.

A high reward-model score is not necessarily a successful task.

The task is successful when the domain’s acceptance criteria say it is successful.

For a coding agent that may mean:

required tests pass
+
no forbidden files changed
+
static checks pass
+
requested behavior exists

For a research agent it may mean:

claims supported by cited sources
+
source quality threshold met
+
contradictions resolved
+
required questions answered

For a browser agent:

expected external state observed
+
confirmation identifier captured
+
no duplicate transaction occurred

The verifier defines the experimental outcome.

Not the agent.

The first benchmark mistake: unequal compute

Suppose we compare these systems:

A: one-shot
B: self-consistency with 8 samples
C: Tree of Thoughts with 8 expansions
D: 3-agent debate + judge

A naive evaluation might simply report success:

one-shot             61%
self-consistency     72%
Tree of Thoughts     75%
debate               73%

That table is almost useless.

The systems consumed different resources.

The useful question is closer to:

What is the best verified success each architecture produces under the same compute envelope?

For example:

budget = 8 model calls per task

Now one-shot is allowed to spend the same budget too.

Maybe it can use:

8 independent solutions + external selection

or:

4 candidates + 4 verifier calls

or:

1 strong model call using a model whose cost matches the budget

That comparison is much harder for advanced orchestration to win.

Good.

That is exactly what you want.

Build a compute envelope

A useful benchmark runner should make the resource envelope explicit.

from dataclasses import dataclass


@dataclass(frozen=True)
class Budget:
    max_model_calls: int
    max_tool_calls: int
    max_input_tokens: int
    max_output_tokens: int
    max_cost_usd: float
    max_seconds: float

Every architecture receives the same budget object.

The runtime owns enforcement.

@dataclass
class Usage:
    model_calls: int = 0
    tool_calls: int = 0
    input_tokens: int = 0
    output_tokens: int = 0
    cost_usd: float = 0.0


def within_budget(usage: Usage, budget: Budget) -> bool:
    return (
        usage.model_calls <= budget.max_model_calls
        and usage.tool_calls <= budget.max_tool_calls
        and usage.input_tokens <= budget.max_input_tokens
        and usage.output_tokens <= budget.max_output_tokens
        and usage.cost_usd <= budget.max_cost_usd
    )

Do not let the architecture quietly exceed the budget because “one more critic call” seemed useful.

That is part of the architecture.

Count it.

Cost per verified success

Raw success often hides the most important trade-off.

Imagine:

Architecture A
success = 70%
average cost = $0.05/task

Architecture B
success = 80%
average cost = $0.50/task

B is more accurate.

But:

A: ~$0.071 per verified success
B: ~$0.625 per verified success

Whether B is better depends on the value and risk of the task.

A useful metric is:

def cost_per_verified_success(total_cost: float, successes: int) -> float:
    if successes == 0:
        return float("inf")
    return total_cost / successes

For high-consequence tasks, paying ten times more may be rational.

For low-value repetitive tasks, it may be absurd.

Architecture evaluation is economic as well as technical.

The second benchmark mistake: changing the verifier

Suppose your simple baseline is judged using exact tests.

Your advanced architecture is judged by an LLM critic.

You no longer have an architecture comparison.

You have changed both the system and the measurement instrument.

The verifier should remain fixed across compared architectures wherever possible.

architecture A ─┐
architecture B ─┼─> same verifier
architecture C ─┤
architecture D ─┘

If the verifier must differ, report that explicitly.

Better yet, separate:

generation architecture
verification architecture

and test them independently.

PASS, FAIL and UNKNOWN

Advanced agents often encounter incomplete evidence.

Do not quietly convert missing evidence into failure.

Do not quietly convert partial evidence into success.

Use three outcomes:

PASS
FAIL
UNKNOWN

For example:

from enum import Enum


class Verdict(str, Enum):
    PASS = "pass"
    FAIL = "fail"
    UNKNOWN = "unknown"

Then report all three rates.

PASS      74%
FAIL      18%
UNKNOWN    8%

An architecture that gets 80% PASS by aggressively guessing through uncertainty may be worse than one that gets 74% PASS and safely returns UNKNOWN for dangerous cases.

Self-consistency: benchmark agreement, not just accuracy

Self-consistency samples multiple reasoning paths and aggregates them.

The obvious metric is final verified success.

But you should also measure:

sample diversity
answer agreement
majority correctness
oracle@N
selection regret

Oracle@N

Oracle@N asks:

Did at least one generated candidate contain a verified correct answer?

def oracle_at_n(verdicts: list[bool]) -> bool:
    return any(verdicts)

If oracle@8 is 92% but final selected success is only 70%, your generator is not the main problem.

Your selector is throwing away good answers.

That distinction matters enormously.

generation problem:
correct answer rarely appears

selection problem:
correct answer appears but is not chosen

Do not add more sampling to fix a bad selector.

Tree of Thoughts: measure pruning regret

Tree of Thoughts introduces partial-state evaluation and pruning.

That creates a new failure mode:

The correct branch existed and you killed it early.

Measure it.

A useful diagnostic is pruning regret.

Conceptually:

branch eventually capable of PASS
        ↓
pruned before completion
        ↓
pruning regret

If you can replay trajectories offline, evaluate pruned branches under a larger diagnostic budget.

Then estimate:

fraction of pruned branches that could have reached PASS

High pruning regret means your evaluator or pruning threshold is weak.

Increasing tree width may help.

But fixing the evaluator may help far more.

Beam search: benchmark width against diversity

Beam search keeps the top-k partial trajectories.

Increasing k does not automatically increase useful exploration.

If all branches are nearly identical, a beam width of 16 may behave like width 1.

Track something like:

unique branch ratio = unique meaningful branches / generated branches

You can define branch uniqueness using:

• normalized action sequences,
• code diff hashes,
• plan structure,
• semantic embeddings,
• tool-call signatures,
• or domain-specific strategy labels.

Then compare:

beam width
unique branch ratio
verified success
cost

You may discover:

width 2   success 71%  uniqueness 0.92
width 4   success 76%  uniqueness 0.78
width 8   success 77%  uniqueness 0.43
width 16  success 77%  uniqueness 0.24

The extra width stopped buying real exploration long before it stopped buying tokens.

MCTS: separate search quality from evaluator quality

MCTS adds a more sophisticated compute-allocation policy.

But MCTS cannot rescue a meaningless value signal.

You should separately evaluate:

selection policy
expansion diversity
rollout quality
value estimate quality
backpropagated value usefulness

A critical comparison is:

MCTS
vs
random expansion with same node budget
vs
beam search with same node budget

If MCTS barely beats random expansion, your UCB machinery is not doing much.

If MCTS beats random but not beam search, the problem may not need exploration/exploitation balancing.

If MCTS wins strongly only when the verifier provides informative intermediate rewards, that tells you exactly where it is useful.

This is the kind of result you want.

Not “MCTS sounds intelligent.”

Evolutionary agents: measure improvement across generations

Evolutionary search often looks impressive because it produces a lot of artifacts.

Track whether generations actually improve.

generation 0 -> verified fitness

generation 1 -> verified fitness

generation 2 -> verified fitness
...

Useful metrics include:

best fitness by generation
median fitness by generation
diversity by generation
novelty
mutation survival rate
crossover survival rate
stagnation length

One especially useful experiment is an ablation:

evolutionary selection
vs
randomly retain same number of candidates

If random retention performs similarly, your fitness function is not steering the population effectively.

Mixture-of-experts routing: benchmark the router separately

Suppose your agent routes tasks among:

small local model
coding specialist
research specialist
frontier model

The full system may look good while the router remains poor.

Measure routing independently.

Useful metrics:

routing accuracy
unnecessary escalation rate
missed escalation rate
cost-weighted routing regret
latency-weighted routing regret
verified success by route

A route can be “wrong” in multiple ways.

Sending an easy task to the frontier model may still succeed.

It is nevertheless economically wrong.

Sending a hard task to the cheap local model may save money but fail.

So routing evaluation should include utility.

For example:

def utility(success: bool, cost: float, latency: float) -> float:
    return (1.0 if success else 0.0) - 0.2 * cost - 0.001 * latency

The coefficients should come from the product’s actual economics, not arbitrary benchmark aesthetics.

Planner / executor / critic systems: ablate each role

If your architecture contains:

planner
executor
critic
verifier

run ablations.

full system
minus planner
minus critic
minus memory
minus search
minus specialist routing

Ask:

What breaks when this component disappears?

If removing the critic changes success from 78% to 78%, the critic is decorative.

If removing it reduces latency by 20%, you have found an obvious simplification.

If removing the planner improves success, your planner is actively harming the system.

Advanced architectures should be removable piece by piece.

Every component should justify itself experimentally.

Multi-agent debate: test diversity, not headcount

Three agents running the same model with the same prompt are not necessarily three independent viewpoints.

They may produce correlated errors.

Track:

initial disagreement rate
error correlation
position-switch rate after debate
correct-to-wrong flips
wrong-to-correct flips
judge accuracy

The most important debate diagnostic may be:

net correction rate
=
wrong -> correct
minus
correct -> wrong

If debate changes ten wrong answers to correct but changes twelve correct answers to wrong, the conversational transcript may look sophisticated while the system gets worse.

Measure the transition matrix.

Adaptive agents: compare adaptation against a fixed policy

An adaptive agent changes its behavior based on task difficulty, uncertainty, history or observed failures.

The right baseline is not a weak one-shot system.

Compare against a fixed policy using the same average resources.

For example:

Adaptive:
2 calls on easy tasks
12 calls on hard tasks
average = 5 calls

Fixed:
5 calls on every task

Then compare verified success and cost.

This reveals whether the adaptation is actually allocating compute intelligently.

Useful adaptive metrics include:

budget allocation by difficulty
escalation precision
escalation recall
unnecessary escalation rate
failure-after-no-escalation rate
marginal gain per extra call

Learning from previous runs: freeze the future

Systems that learn from prior trajectories create a special evaluation risk.

If you tune policies using the same tasks you later report as evidence, you have leaked the benchmark.

Use temporal or task-family splits.

For example:

runs 1-1000
    ↓
learning / calibration

runs 1001-1300
    ↓
frozen evaluation

Or:

repository families A/B/C
    ↓
learning

repository family D
    ↓
evaluation

The important rule is:

The benchmark must contain future information the learning system did not already absorb.

Otherwise you are measuring memory, not generalization.

The benchmark dataset should contain failures

Do not build a benchmark made only of tasks your agent already solves.

You need a useful difficulty distribution.

For example:

simple tasks
moderate tasks
hard tasks
ambiguous tasks
adversarial tasks
unverifiable tasks

Why include unverifiable tasks?

Because a production agent needs to know when to stop pretending.

A good system should sometimes return:

UNKNOWN

instead of hallucinating certainty.

Stratify by failure mode

Overall success can hide what changed.

Label tasks by the problem they stress.

For coding agents:

local syntax fix
cross-file change
API migration
ambiguous requirement
failing test diagnosis
performance regression
concurrency bug
repository exploration

Then report success by bucket.

You may find:

self-consistency helps ambiguous requirements
Tree of Thoughts helps repository exploration
MCTS helps long multi-step debugging
specialist routing helps API migration
none of them help trivial syntax fixes

That result is far more actionable than one global score.

Measure marginal gain

Advanced agents should justify the next unit of compute.

Define:

marginal gain = success(B + ΔB) - success(B)

Then plot or tabulate:

budget    success
1 call     61%
2 calls    68%
4 calls    73%
8 calls    76%
16 calls   77%
32 calls   77.5%

The curve tells you where extra computation stops paying.

A production runtime can use this information directly.

cheap path
   ↓ if unresolved
moderate path
   ↓ if unresolved
expensive path
   ↓
UNKNOWN / human escalation

This is far more useful than configuring every task with the maximum architecture.

Confidence intervals matter

Suppose architecture A scores 78% and B scores 80% on 50 tasks.

That does not mean B is better.

The difference may be noise.

At minimum, report uncertainty.

For binary outcomes, bootstrap confidence intervals are easy to compute.

import random


def bootstrap_success_interval(results, samples=5000):
    means = []
    n = len(results)

    for _ in range(samples):
        sample = [random.choice(results) for _ in range(n)]
        means.append(sum(sample) / n)

    means.sort()
    low = means[int(0.025 * samples)]
    high = means[int(0.975 * samples)]
    return low, high

Even better, when architectures run on the same tasks, use paired comparisons.

The question becomes:

On which exact tasks did B improve over A?
On which exact tasks did it regress?

That often reveals the mechanism far better than the aggregate.

Keep the task order deterministic

Agent systems are stochastic.

Benchmarks should remove avoidable randomness.

Record:

task ID
architecture version
model version
prompt version
seed where supported
tool versions
repository/data revision
verifier version
budget

A benchmark result without configuration provenance becomes hard to reproduce almost immediately.

Cache carefully

Caching can make advanced-agent benchmarks dramatically cheaper.

It can also invalidate comparisons.

If architecture A receives cached model outputs and B does not, your latency comparison is meaningless.

Separate:

logical model calls
physical paid model calls
cache hits

For architecture quality, logical calls usually matter.

For operational economics, physical calls matter.

Report both.

A minimal benchmark record

A useful run record might look like this:

from dataclasses import dataclass, field
from typing import Any


@dataclass
class BenchmarkResult:
    task_id: str
    architecture: str
    verdict: str
    model_calls: int
    tool_calls: int
    input_tokens: int
    output_tokens: int
    cost_usd: float
    latency_seconds: float
    verifier_version: str
    state_id: str
    metadata: dict[str, Any] = field(default_factory=dict)

Do not store only the final score.

Store enough detail to explain the score later.

Compare trajectories, not only answers

Two systems can both pass while one is far healthier.

Consider:

System A
12 tool calls
4 failed actions
3 repeated actions
1 recovered final result
PASS

System B
4 tool calls
0 failed actions
PASS

Both count as successful.

But their trajectories differ substantially.

Track operational metrics such as:

repeated-action rate
invalid-action rate
recovery count
replan count
branch count
pruned nodes
rollback count
unsafe-action rejection count

These metrics often predict production reliability before final success rate changes visibly.

Benchmark real software classes differently

The benchmark framework is generic.

The verifier is not.

Coding agents

Use evidence such as:

tests
static analysis
build result
diff constraints
behavioral checks
repository state

Useful architecture diagnostics:

oracle@N
patch selection regret
compile-before-test rate
test-failure reduction
unnecessary file edits
cost per accepted patch

Research agents

Use:

claim-source coverage
source authority
citation correctness
contradiction detection
freshness

Useful diagnostics:

source diversity
unsupported-claim rate
retrieval redundancy
search-query efficiency
cost per verified claim

Browser agents

Use:

external page state
transaction identifiers
form validation
account state
idempotency evidence

Track:

navigation loops
repeated form submission
stale DOM actions
recovery rate
verified transaction success

Data agents

Use:

schema checks
row counts
invariants
reconciliation
sample validation

Track:

invalid transformations
rollback rate
pipeline retries
data-loss indicators
cost per validated transformation

DevOps agents

Use:

deployment state
health checks
metrics
logs
rollback status
incident acceptance criteria

Track:

unsafe action rejection
rollback success
mean actions to recovery
false remediation rate
verified incident resolution

Do not benchmark only happy paths

Inject failure.

Examples:

tool timeout
malformed tool result
stale memory
wrong retrieved document
partial repository checkout
failing verifier
contradictory evidence
rate limit
model refusal
unexpected environment state

Then measure what the agent does.

A robust architecture should not only succeed more often.

It should fail better.

That means:

fewer unsafe actions
more explicit UNKNOWN states
better recovery
less repeated work
clearer evidence trails

A practical benchmark matrix

Imagine testing four architectures:

A = one-shot + verifier
B = Best-of-N + verifier
C = beam search + verifier
D = MCTS + verifier

Run each under budgets:

2 calls
4 calls
8 calls
16 calls

Now you can build a surface rather than a single ranking.

             2      4      8      16
one-shot    61%    64%    65%    65%
Best-of-N   64%    70%    74%    76%
beam        62%    71%    77%    79%
MCTS        60%    69%    80%    84%

This tells a much richer story.

At tiny budgets, MCTS may be wasteful.

At larger budgets, it may allocate computation better.

The right production architecture may therefore be adaptive:

easy task -> Best-of-N
hard task -> beam
very hard task with strong intermediate reward -> MCTS

The benchmark becomes architecture policy.

The strongest baseline may be a better model

Before celebrating a 20-call orchestration system, compare it against spending the same money on a stronger model.

For example:

20 cheap-model calls
vs
4 medium-model calls
vs
1 frontier-model call

Sometimes orchestration wins.

Sometimes the stronger single model destroys the elaborate architecture.

Either result is useful.

Your goal is not to prove agent complexity is valuable.

Your goal is to discover what is valuable.

Benchmark the simplest deterministic solution too

There is one more baseline advanced-agent work often forgets:

deterministic software

If the task is fully specified and the route is known, a workflow engine or normal program may outperform every agent architecture in:

accuracy
latency
cost
reproducibility
security

Always ask:

Does this task require adaptive reasoning at all?

Sometimes the best agent benchmark result is evidence that you should delete the agent.

Build an architecture leaderboard carefully

A useful leaderboard should not rank only by success.

For example:

architecture        success  cost/success  p95 latency  false-pass  unknown
one-shot             64%       $0.04          3s          3%         2%
Best-of-N            74%       $0.11          8s          2%         3%
beam                  79%       $0.18         14s          1%         4%
MCTS                  82%       $0.33         31s          1%         5%
mixture               81%       $0.15         12s          1%         4%

Now architecture choice becomes a product decision.

A low-latency assistant may prefer Best-of-N.

A high-value offline coding task may prefer MCTS.

A mixed production workload may prefer a router that escalates selectively.

No single number decides everything.

Benchmark architecture versions, not names

“MCTS” is not one system.

Neither is “Tree of Thoughts” or “multi-agent debate”.

Record configuration.

mcts-v3
exploration_c=1.2
max_nodes=64
rollouts=4
value_model=scorer-v7
verifier=test-suite-v12
model=qwen-x

Small configuration changes can matter more than the family name.

Treat agent architectures as software releases.

Version them.

The complete experiment loop

A mature process looks like this:

identify production failure
        ↓
construct benchmark cases
        ↓
freeze verifier
        ↓
run simple baseline
        ↓
introduce one mechanism
        ↓
match compute budget
        ↓
measure verified success + cost + latency
        ↓
inspect per-task wins and regressions
        ↓
ablate mechanism
        ↓
failure injection
        ↓
keep / tune / remove

This is the difference between agent architecture and agent decoration.

Ten rules for benchmarking advanced agents

1. Use external verification. Agent confidence is not ground truth.
2. Match compute budgets. More calls are not free architectural intelligence.
3. Compare against stronger-model baselines. Orchestration must beat alternative ways to spend the same money.
4. Include deterministic software where appropriate. Not every task needs an agent.
5. Track oracle@N. Separate generation failures from selection failures.
6. Ablate components. Every planner, critic, router and memory layer must earn its place.
7. Measure cost per verified success. Raw success can hide terrible economics.
8. Stratify by failure mode. Global averages hide where mechanisms actually help.
9. Preserve UNKNOWN. Safe uncertainty is often better than false success.
10. Keep only mechanisms that produce repeatable marginal gain. Complexity is not the objective.

The deeper lesson

Advanced agent systems are tempting because every extra mechanism has an intuitive story.

A critic should catch mistakes.

Search should find better paths.

Debate should reveal weaknesses.

MCTS should allocate compute intelligently.

Specialists should outperform generalists.

Memory should prevent repeated failures.

Adaptation should make the runtime smarter over time.

All of those statements can be true.

They can also be false in your application.

The architecture does not get credit for sounding plausible.

It gets credit for measurable improvement.

The final rule is therefore the same one that has run through this entire series:

Start with the simplest system that can work. Identify a measurable failure. Add the smallest mechanism that targets that failure. Benchmark it under equal resources. Keep it only if verified outcomes improve enough to justify the cost.

That is how advanced agents become engineering rather than mythology.