The previous chapter built a boundary that stops arbitrary text from becoming an action. It solved one class of failure completely, and it is silent about another.

Suppose the model is asked to solve a coding problem. One run produces the right patch. The next produces a plausible but incomplete one. A third produces something better again. Nothing is malformed, nothing violates the action schema, and every one of them would pass the boundary we just built. Validity and quality are different properties. A proposal can be completely valid and still be a poor choice, which relocates the uncertainty rather than removing it:

Which valid proposal should we keep?

The mechanism this chapter adds is the simplest possible answer: stop accepting the first proposal, generate several, and make the choice between them an explicit part of the runtime. This is usually called Best-of-N.

The model does not change. We spend more inference-time computation to create alternatives, and we spend engineering effort on the selection step that was previously invisible because there was nothing to select from. And the reason this is worth a chapter rather than a paragraph is the split it creates:

Separate the ability to generate a good candidate from the ability to recognise one.

Those are different capabilities, they fail independently, and once they are separate they become measurable. That distinction is the entire chapter.

1. Start with the one-shot baseline

Our current system makes exactly one proposal:

def answer(prompt, llm):
    return llm(prompt)

This should stay the baseline, and it should stay the default. Do not add Best-of-N because several model calls look more sophisticated — add it because the baseline has a failure you can point at, such as output quality varying too much across the same task distribution.

So record the baseline before changing anything. For each task, keep task_id, success, latency, model_calls and tokens. Those five fields decide whether this chapter applies to you at all.

If one call already solves almost every task, sampling eight answers buys cost and latency and nothing else. If one call is unreliable but repeated attempts often contain a good solution, Best-of-N has something real to exploit. That second condition is doing all the work, and it is worth measuring rather than assuming.

2. Best-of-N is horizontal search

Generate several complete candidates from the same starting point:

def generate_candidates(prompt, llm, n=4):
    return [llm(prompt) for _ in range(n)]

    flowchart LR
    P[prompt] --> M[model]
    M --> A[candidate A]
    M --> B[candidate B]
    M --> C[candidate C]
    M --> D[candidate D]
    A --> S[evaluate<br/>and select]
    B --> S
    C --> S
    D --> S
  

This is the smallest search mechanism in the book, and it is worth naming its exact shape: Best-of-N is horizontal search over complete proposals at a single decision point.

That precision matters because two later chapters build things that look similar and behave differently. Critique and revision produces vertical depth — one candidate refined repeatedly against feedback. Trajectory search explores partial states, branching mid-solution rather than at the start. Best-of-N does neither. It samples complete alternatives from one state and picks one, which is exactly why it belongs this early: it requires no planner, no memory, no critic and no tree.

3. Two jobs, and the instrument that separates them

Suppose we generate four candidates and, after the fact, we can tell which were actually correct:

Both cases produce a wrong final answer. They call for completely different work — one week spent on prompt diversity, one week spent on the evaluator — and a single success number cannot tell them apart. So we measure two things.

oracle@N asks whether the candidate set contained an eligible candidate that would satisfy the external success criterion. It measures the opportunity the generator created for the selector. Eligibility matters. For ordinary text candidates, every generated candidate may be eligible. For executable actions, the previous chapter’s boundary applies first: a malformed, unauthorized or currently unexecutable action cannot count as selector opportunity merely because it would otherwise look successful.

selection_gap@N asks how much of that opportunity the selector wasted:

selection_gap@N = oracle@N − selected_success@N

Both need an external success criterion — unit tests for code, a known answer for a numerical problem, a ground-truth fixture for structured data. Given one, the whole instrument is about fifteen lines:

from dataclasses import dataclass
from statistics import mean


@dataclass
class Candidate:
    value: str
    score: float | None = None
    eligible: bool = True
    verified_success: bool | None = None


@dataclass
class RunRecord:
    task_id: str
    candidates: list[Candidate]
    winner_index: int

    @property
    def oracle_success(self) -> bool:
        """Did the generator produce an eligible successful candidate?"""
        return any(
            c.eligible and c.verified_success
            for c in self.candidates
        )

    @property
    def selected_success(self) -> bool:
        """Did the selector return an eligible successful candidate?"""
        winner = self.candidates[self.winner_index]
        return winner.eligible and bool(winner.verified_success)


def oracle_at_n(runs: list[RunRecord]) -> float:
    return mean(r.oracle_success for r in runs)


def selected_at_n(runs: list[RunRecord]) -> float:
    return mean(r.selected_success for r in runs)


def selection_gap(runs: list[RunRecord]) -> float:
    return oracle_at_n(runs) - selected_at_n(runs)

Now run it across a sweep of N and read the two curves together:

Read oracle@N alone and the system looks like it is improving. Read selected success and improvement stops after N=4 before declining. Read the selection gap and the diagnosis becomes unambiguous: at N=8 the generator is producing a successful option 89% of the time, while the selector is failing to capture sixteen percentage points of that available opportunity.

A large gap says stop trying to make the generator more creative; the answers are already there; fix selection. A small gap with low oracle success says the opposite: the selector is doing about as well as the candidate set permits, so improve generation. This is the main debugging instrument of the chapter, and later chapters reuse its shape for tool routing, memory recall and search pruning.

Generating many candidates and using a separate verifier to rank them is a well-established pattern; Cobbe et al. generated many mathematical solutions and trained a verifier to select among them, which is a particularly clean instance of treating generation and selection as different capabilities.[1]

4. Build the smallest Best-of-N system

The mechanism itself needs a generator, an evaluator and an argmax:

from typing import Callable


def best_of_n(
    prompt: str,
    generate: Callable[[str], str],
    score: Callable[[str, str], float],
    n: int = 4,
) -> tuple[Candidate, list[Candidate]]:
    if n < 1:
        raise ValueError("n must be at least 1")

    candidates = [Candidate(value=generate(prompt)) for _ in range(n)]

    for candidate in candidates:
        candidate.score = score(prompt, candidate.value)

    winner = max(candidates, key=lambda c: c.score)
    return winner, candidates

Note that it returns every candidate, not only the winner. That is not a convenience — it is what makes §3’s instrument computable at all. A function that returns only the winner has destroyed the evidence needed to tell case 1 from case 2.

5. What should evaluate the candidates?

The selector is only as good as the signal it uses, and there is no single best evaluator. There is, however, a useful order of preference: external task evidence where it exists, then a deterministic task-specific metric, then a learned scorer with validated behaviour, then an LLM judge with measured limitations.

This is not an absolute hierarchy — some tasks genuinely require subjective judgement. It exists to prevent one specific mistake: asking a language model to judge something the environment can measure directly.

Direct task evidence is usually the strongest evaluator when the task provides it. If code has tests that genuinely measure the required behaviour, run them. If a numerical answer has an exact oracle, compute it. If a hard constraint can be checked deterministically, check it. This does not make the evaluator infallible. Tests can be incomplete and benchmarks can be weak proxies. Chapter 09 will deal with that verification problem directly. For selection, the immediate rule is narrower:

Prefer evidence about the task over a model’s opinion about the task when that evidence is available.

def score_patch(patch: str) -> float:
    result = run_tests(patch)
    return result.passed / result.total

If an answer can be checked numerically, check it. If a required constraint can be checked directly, check it.

Deterministic metrics work when the target property is directly computable — latency, constraint coverage, output size, or any other property that is a function of the candidate rather than an opinion about it:

def score_constraint_coverage(candidate) -> float:
    return sum(check(candidate) for check in required_checks)

A learned scorer fits where the property is real but not directly computable. The runtime needs nothing from it but a stable interface — score = scorer(prompt, candidate) — which means the scorer can be replaced or retrained without touching the agent.

An LLM judge is the last resort and is genuinely useful for open-ended language tasks. It is also a probabilistic evaluator rather than ground truth, which the next two sections take seriously.

6. Pointwise and pairwise selection are different decisions

A pointwise scorer evaluates each candidate independently and we take the maximum. A pairwise selector instead asks which is better, A or B?, which helps when absolute scores are hard to calibrate but relative preference is easy to express.

The simplest tournament is a fold:

def tournament(candidates, prefer):
    winner = candidates[0]

    for challenger in candidates[1:]:
        winner = prefer(winner, challenger)

    return winner

That code hides two properties worth knowing about. The result can depend on comparison order, because a fold is not order-invariant unless prefer is a genuine ordering. And preferences may be cyclic — A over B, B over C, C over A — in which case no ordering exists at all and the tournament simply reports whichever cycle entry it happened to end on.

So pairwise does not mean objective. It changes the form of the decision, not its reliability.

7. LLM judges are components to be measured, not oracles

Give a judge candidates A and B and it chooses A. Reverse the order and present B first. If the preference flips, the evaluator is responding to position rather than quality — and the research literature on LLM-as-a-judge has documented position, verbosity and self-enhancement biases among other limitations.[3]

This is cheap to detect. Evaluate both orders and record the agreement:

def symmetric_preference(a, b, judge):
    forward = judge(a, b)
    reverse = judge(b, a)

    if forward == "A" and reverse == "B":
        return a

    if forward == "B" and reverse == "A":
        return b

    return None

The None is the important part of that function. It means the evaluator did not establish a stable preference under this control, so the runtime preserves that uncertainty instead of inventing a winner. Judge disagreement is a measurement, and it tells you something real about how far to trust this selector on this task. Collapsing it into a coin flip converts evidence into false certainty, and the resulting number will look exactly like a confident decision in every log you keep afterwards.

8. Candidate count and candidate diversity are different variables

Four generations do not necessarily give you four ideas:

A: use a dictionary cache
B: cache the result in a dictionary
C: use a dict as a cache
D: store previous results in a dictionary

That is four samples and roughly one strategy. oracle@N will barely move, because sampling explored the same region four times.

Diversity can be inspected with exact duplicate rate, lexical similarity, embedding similarity, or explicit strategy labels — which metric fits depends on the task. The question they all approximate is the same one:

Did additional sampling explore a meaningfully different part of the solution space?

The obvious lever is temperature, and it is worth being careful with it. Raising temperature may produce genuinely different approaches; it may equally produce more low-quality and invalid candidates, trading baseline quality for spread. Treat it as an experimental parameter with a measurable effect on both curves rather than as a cure for homogeneity.

A more controllable option is to request diversity structurally:

strategies = [
    "Solve directly and minimally.",
    "Look for edge cases first.",
    "Try a data-structure-oriented solution.",
    "Try a correctness-first solution.",
]


def generate_diverse(prompt, llm):
    return [llm(f"{prompt}\n\nApproach: {strategy}") for strategy in strategies]

Now the alternatives differ because the runtime asked for different approaches, not because decoding was noisier. The diversity is a property of the design rather than a side effect of sampling.

9. More candidates can produce a worse selected answer

This one is easy to miss, and it is the reason §3 measures two curves instead of one.

Suppose a scorer has a slight preference for confident, verbose answers even when they contain subtle errors. At N=2, neither candidate is likely to exploit that weakness. At N=32, the generator has thirty-two chances to produce something that scores extremely well against the proxy while being worse against the real task. Increasing N can improve the best candidate available while degrading the candidate actually selected. The larger candidate set creates more genuine opportunity, but it also creates more opportunities to produce an answer that exploits whatever the evaluator rewards imperfectly. This is proxy overoptimisation, and it is not hypothetical. Gao, Schulman and Hilton studied it directly for reward models including Best-of-N sampling: as optimisation pressure against an imperfect proxy increases, proxy reward and true quality diverge.[4]

Never assume selected quality rises monotonically with N. Measure the curve.

The table in §3 shows exactly this shape — oracle still climbing at N=8 while selected success has already turned over. Sweep N across 1, 2, 4, 8 and 16 and record both.

10. What the extra inference actually buys

Best-of-N does not make the base model smarter. It changes how much computation is spent around one decision, and the bill arrives in several parts. In the simplest unbatched implementation, eight generated candidates plus eight pointwise LLM judgements require sixteen model invocations. A pairwise tournament adds another N − 1 comparisons. Symmetric judging, from §7, doubles those again.

So the metrics that matter are not just accuracy: track selected success, oracle@N, selection gap, model calls, tokens, latency, and cost per successful task. That last one is the honest denominator, and it frequently tells a different story from raw accuracy.

Recent work on test-time compute makes the same broader point from the research side: more inference-time computation can improve performance, but the useful allocation depends on the problem and the method, and simply increasing a fixed Best-of-N budget is not automatically compute-optimal.[5]

What extra success did the extra inference buy?

11. The action boundary still applies to every candidate

When Best-of-N selects actions rather than final text, the contract from the previous chapter does not go away — it runs N times.

Suppose four candidate actions come back: one valid, one with a schema error, one unauthorised, one valid. Ranking all four as equivalent candidates is a category error, because two of them cannot execute at all. The boundary comes first, and only accepted actions reach the ranker:

    flowchart LR
    R[raw candidates] --> B{complete action boundary}
    B -->|eligible| A[eligible candidate set]
    B -.rejected, by stage.-> T[rejection counts]
    A --> K[rank]
    K --> E[execute]
  

In code, reusing the Decision union and Stage enum from the previous chapter:

from collections import Counter


def accepted_candidates(
    raw_candidates,
    policy,
    context,
) -> tuple[list[Action], Counter]:
    accepted, rejected = [], Counter()

    for raw in raw_candidates:
        decision = prepare_action(raw, policy, context)

        if isinstance(decision, Accepted):
            accepted.append(decision.action)
        else:
            rejected[decision.stage] += 1

    return accepted, rejected

Returning the rejection counter costs one line and pays for itself immediately. A run where six of eight candidates fail at Stage.SCHEMA is a prompt or schema problem wearing a selection problem’s clothes, and without that counter it looks identical to a run where all eight were valid and the selector chose badly.

For executable candidates, ranking therefore happens only after the complete action boundary:

representation
→ schema
→ semantics
→ authorization
→ preconditions
→ eligible candidate set
→ ranking

12. Neighbouring mechanisms, and how they differ

Three techniques in this space get used interchangeably in conversation and are not interchangeable in practice:

Self-consistency is the interesting near-neighbour. It samples multiple reasoning paths as Best-of-N does, then aggregates agreement over the resulting answers rather than ranking each complete candidate with a separate quality score. For tasks with extractable discrete answers, that often reduces to selecting the answer supported by the largest share of sampled paths. That works when answers are comparable and agreement correlates with correctness, and it fails silently when the model is confidently and consistently wrong. Critique and revision therefore needs feedback about the defect and evidence for whether the attempted correction actually improved the candidate. The critic proposes what is wrong; it does not get to decide that its own repair succeeded.

    flowchart TB
    subgraph BoN[Best-of-N: breadth]
        direction LR
        P[prompt] --> A1[A] & B1[B] & C1[C]
        A1 & B1 & C1 --> S[select]
    end
    subgraph CR[Critique and revision: directed depth]
        direction LR
        P2[prompt] --> A0[A0] -->|critique| A2[A1] -->|critique| A3[A2]
    end
  

One explores breadth from a fixed starting point. The other adds directed depth to a single line of work. The next chapter builds the second.

13. A useful experiment

The cleanest test of this chapter holds the base model fixed and varies only N. For a controlled pilot, take a small task set with external ground truth — twenty short Python functions with unit tests is enough to make the mechanism visible — and run N at 1, 2, 4 and 8, storing every candidate rather than only the winner. Do not treat twenty tasks as a production benchmark. The purpose here is to expose the shape of the curves and verify that the diagnostic behaves as expected before scaling the evaluation.

Then compute oracle@N, selected success, selection gap, candidate diversity, model calls, tokens, latency and cost per success, and ask four questions in order:

1. Does oracle@N increase? If not, additional generation is producing no new opportunity — the samples are redundant, and §8 is where to look.
2. Does selected success track oracle@N? If not, the selector is leaving good candidates behind and §5 through §7 are where to look.
3. Does the selection gap grow with N? If so, increased search is exposing weaknesses in the evaluator faster than it is creating opportunity — the §9 failure.
4. Was the improvement worth the compute? If not, use a smaller N, or go back to one-shot.

Each question routes to a different section of this chapter, which is the property that makes the experiment worth running. Best-of-8 scored 76% routes nowhere.

14. Telemetry that preserves the diagnosis

All of the above depends on keeping the evidence, and the RunRecord from §3 already is the trace. Serialising it is enough:

record = {
    "task_id": run.task_id,
    "n": len(run.candidates),
    "winner_index": run.winner_index,
    "rejected_by_stage": dict(rejected),
    "candidates": [
        {
            "value": c.value,
            "score": c.score,
            "eligible": c.eligible,
            "verified_success": c.verified_success,
        }
        for c in run.candidates
    ],
}

From that one record you can reconstruct, weeks later, whether a correct candidate was available, which one the selector chose, how large the score margin was, whether the candidates were near-duplicates, and whether the chosen answer was actually right.

Store only the final answer and every one of those questions becomes unanswerable. This is the most common way teams end up unable to explain their own agent: the evidence was discarded at the moment it was cheapest to keep.

15. The deeper lesson

Each chapter so far has taken a decision that was happening invisibly and given it a name, a contract and a measurement. The first separated a model call from an adaptive loop. The second separated a proposal from executable authority. This one separates generation from selection.

The addition was never “four model calls.” It was a new place to reason about failure. A bad result can now be attributed to poor candidate generation, insufficient candidate diversity, a weak evaluator, a selection rule that mishandles ties or cycles, or proxy overoptimisation — five distinct defects that used to be one undifferentiated the agent got it wrong.

Expose the hidden decision, give it a contract, and measure the mechanism that owns it.

That is the pattern the rest of the book repeats, and §3’s two-curve instrument is the first fully general instance of it. When we get to tool routing, memory recall and search pruning, we will be asking the same question in different clothes: was the right thing available, and did we pick it?

Research roots

The chapter is built from first principles, but several research lines establish the ancestry and failure modes of the mechanism.

1. Cobbe et al. — Training Verifiers to Solve Math Word Problems (2021). Generate many candidate solutions and train a verifier to rank them, providing a particularly clean example of generation and selection as separate capabilities. https://arxiv.org/abs/2110.14168
2. Wang et al. — Self-Consistency Improves Chain of Thought Reasoning in Language Models (ICLR 2023). Samples diverse reasoning paths and aggregates answer agreement; related to Best-of-N but useful precisely because its selection rule is different. https://arxiv.org/abs/2203.11171
3. Zheng et al. — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023). Studies LLM judges and documents limitations including position, verbosity and self-enhancement biases. https://arxiv.org/abs/2306.05685
4. Gao, Schulman & Hilton — Scaling Laws for Reward Model Overoptimization (2022/2023). Shows that optimizing an imperfect reward proxy, including through Best-of-N sampling, can eventually separate proxy score from true quality. https://arxiv.org/abs/2210.10760
5. Snell et al. — Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (2024). Studies inference-time compute allocation and compares compute-aware methods against fixed Best-of-N baselines. https://arxiv.org/abs/2408.03314

Next: Critique, Revision, and Acceptance

Best-of-N gives the system breadth. It can try A, B, C and D and keep the best one. What it cannot do is ask why A failed, or what B would need in order to work.

That is the next missing mechanism. When the first candidate is close but wrong, throwing it away and sampling another independent answer discards the most useful thing we have: specific evidence about a specific defect. The next chapter inspects that defect, makes a targeted revision, compares the revision against the original, and — crucially — decides whether to keep it. That last step is where most implementations of this idea go wrong.

Best-of-N asked which complete proposal to keep. The next chapter asks whether evidence about a proposal’s failure can tell us how to improve it.