Chapter 01 of 12

Reasoning Is More Than Architecture — Where Extra Computation Lives

Explain this chapter with AI

Copy this prompt into ChatGPT, Claude, Gemini, a local model, or another AI.

Apply this chapter with AI

Copy this prompt into ChatGPT, Claude, Gemini, a local model, or another AI.

Reasoning Is More Than Architecture — Where Extra Computation Lives

So far in Models From First Principles, we have changed several different things and called all of them model design.

We changed what a model predicts.

MR.Q produced one learned quality score.

EBT added Q, V, policy, and advantage.

SICQL turned those outputs into explicit model components.

Then we changed how computation unfolds.

HRM introduced recurrent state operating at different timescales.

Tiny kept repeated computation while shrinking the architecture.

Inside Tiny, we opened that recurrence and found residual blocks, attention, sparse representations, and output heads.

Then we changed how parameters move.

PACS moved underneath the forward pass into gradient statistics and optimizer state.

And in the previous chapter, we changed the supervision itself.

Preference Rankers replaced awkward absolute targets with pairwise comparisons.

At this point a larger pattern becomes visible.

Model capability is not determined by architecture alone.

A system can change because we altered its parameters.

It can change because we gave the same parameters more recurrent computation.

It can change because we sampled the same model several times instead of once.

It can change because a verifier or ranker selected among those samples.

It can change because post-training altered which behaviours the parameters prefer.

It can change because expensive trajectories were distilled into a cheaper model.

Those are different mechanisms.

They spend computation in different places.

They fail in different ways.

And if we collapse all of them into the phrase reasoning model, we lose the thing this book has been trying to preserve from the beginning:

the mechanism.


1. Start with a better question

When a model performs better on a reasoning task, do not begin with:

What new architecture did they invent?

Begin with:

Where did the additional capability come from?

A useful first-principles map is:

                         observed performance
          ┌─────────────────────┼─────────────────────┐
          │                     │                     │
          ▼                     ▼                     ▼
    model structure        inference process      training process
          │                     │                     │
    parameters             recurrent steps         objectives
    attention              more samples            preference data
    recurrence             voting                  reinforcement learning
    hierarchy              ranking                 distillation
    representations        verification            optimizer dynamics

The categories overlap.

A recurrent model changes architecture and inference cost.

A preference ranker may be used during training or during inference-time selection.

Distillation changes training so that expensive behaviour can become cheaper at inference.

But the map gives us a much better starting point than treating every improvement as a new neural architecture.


2. Architecture asks what computation exists

The first half of this book focused heavily on architecture.

For MR.Q:

context + candidate
      encoder
      score

For SICQL:

representation
      ├──→ Q
      ├──→ V
      └──→ Policy

For Tiny:

context + candidate + latent
         reusable core
      latent state update
           repeat

These changes alter the computation graph itself.

They change which tensors exist, which parameters are learned, which state persists, and how information flows.

That is architecture.

But architecture is only one place we can spend additional compute.


3. Parameter count and computation are different resources

Tiny already gave us the first clue.

Suppose a block has one million parameters.

Run it once:

1M parameters
1 application

Run the same block eight times:

1M parameters
8 applications

The parameter count did not change.

The computation did.

This distinction is fundamental:

parameter capacity
computation depth

A recurrent model can spend more compute without adding a fresh parameter set for every step.

Likewise, an ordinary autoregressive language model can spend more inference compute by generating more tokens or more candidate trajectories without changing its weights at all.

So when someone says a system has more reasoning, one of the first questions should be:

Did the parameters change, or did we simply use the parameters differently?


4. One trajectory can become longer

Tiny spends more computation along a single state trajectory:

z0
core
z1
core
z2
core
z3

This is sequential scaling.

Each later state depends on the state before it.

The model gets additional opportunities to transform an intermediate representation before producing the final output.

Autoregressive reasoning can do something structurally similar in token space:

prompt
partial trajectory
longer trajectory
longer trajectory
answer

The mechanisms are not identical.

Tiny updates latent vectors through a recurrent neural block.

A language model generates tokens autoregressively.

But both spend more compute by extending one path.

That gives us our first inference-time dimension:

one trajectory
more steps

5. Or one trajectory can become many

There is another way to spend inference compute.

Instead of making one path longer, sample several complete candidates:

                    ┌──→ candidate A
prompt → same model ├──→ candidate B
                    ├──→ candidate C
                    └──→ candidate D
                           reducer
                         final answer

The model parameters are identical in every branch.

What changed is the inference procedure.

We paid for several trajectories and then added a rule for reducing them to one result.

This is a different computational object from a single model call.

It is better represented as:

system = model + sampling policy + reduction rule

That reduction rule might be:

  • majority vote;
  • self-consistency;
  • a learned ranker;
  • a deterministic verifier;
  • a reward model;
  • a human choice;
  • a domain-specific test.

The selector is not a detail.

It is part of the system.


6. Sampling does not automatically create intelligence

Suppose one model call gives a correct answer with probability:

p = 0.60

Calling the model eight times does not mean the system is now:

8 × 0.60 = 4.8 times correct

That expression is meaningless.

The samples may be highly correlated.

They may all fail in the same way.

They may disagree for superficial reasons.

And even if one candidate is correct, the system still needs a way to identify it.

This gives us a recurring rule:

Generation capacity and selection capacity are different capabilities.

We saw the same distinction earlier with Preference Rankers.

A generator produces alternatives.

A ranker decides between alternatives.

Do not assume that improving one automatically improves the other.


7. Self-consistency: sample several paths, reduce by agreement

One simple strategy is self-consistency.

Generate several reasoning trajectories and extract their final answers.

For example:

trajectory 1 → 42
trajectory 2 → 42
trajectory 3 → 41
trajectory 4 → 42
trajectory 5 → 39

Then choose the most common result:

42

The important mechanism is not the phrase self-consistency.

It is:

sample diverse paths
extract comparable outcomes
aggregate outcomes

A tiny implementation is ordinary Python:

from collections import Counter


def majority_vote(values):
    counts = Counter(values)
    return counts.most_common(1)[0][0]

The difficult part is not the Counter.

The difficult parts are:

  • generating sufficiently diverse trajectories;
  • extracting the answer correctly;
  • deciding whether several textual answers are actually equivalent;
  • handling ties;
  • choosing the sample budget;
  • knowing whether agreement correlates with correctness.

The infrastructure around the model determines whether the method is useful.


8. Agreement is not verification

Suppose five trajectories produce:

A
A
A
A
B

Majority vote strongly prefers A.

But all four A trajectories may share the same mistake.

Agreement tells us:

what the sampled distribution concentrates on

It does not necessarily tell us:

what is true

This distinction matters enormously.

A majority vote is a reduction rule.

A verifier is an evidence mechanism.

Sometimes they coincide well enough to be useful.

Sometimes they do not.


9. Best-of-N: generate first, score second

Another common pattern is Best-of-N.

Generate several candidates:

c1
c2
c3
...
cN

Then score them:

score(c1)
score(c2)
score(c3)
...
score(cN)

and choose:

argmax score(ci)

In code, the orchestration is tiny:

def best_of_n(generate_one, score, prompt, n=8):
    candidates = [
        generate_one(prompt)
        for _ in range(n)
    ]

    scored = [
        (score(prompt, candidate), candidate)
        for candidate in candidates
    ]

    return max(scored, key=lambda item: item[0])

Again, the loop is not the interesting part.

The real question is:

What is score?

It might be MR.Q.

It might be a Preference Ranker used pairwise.

It might be a deterministic unit test.

It might be a mathematical verifier.

It might be another language model.

Those choices produce very different systems.


10. The selector can become the bottleneck

Imagine the generator produces ten candidates and one is genuinely excellent.

But the selector consistently prefers a fluent wrong answer.

Increasing N can make the system worse.

Why?

Because a larger candidate pool gives the selector more opportunities to choose something it systematically overrates.

That means test-time scaling can expose a weakness that was previously hidden:

small N
generator bottleneck

large N
selector bottleneck

This is the same lesson we learned throughout the book:

A larger system is only as useful as the interfaces between its components.


11. A preference ranker can move from training to inference

The previous chapter treated preference ranking primarily as a learned comparison mechanism.

Now we can reuse the same idea during inference.

Suppose the generator produces:

candidate A
candidate B
candidate C
candidate D

We can compare candidates pairwise:

A vs B → A
A vs C → C
C vs D → C

and retain C.

The architecture of the ranker has not changed.

Its role has.

This is a useful reminder:

The same model can participate in different systems depending on where it is placed in the computation graph.

A learned component does not have one universal meaning.

Its meaning comes partly from its interface and use.


12. A verifier changes the problem again

For some domains we can do better than preference.

Suppose a candidate claims that a Python function is fixed.

We can run the tests.

Suppose a candidate gives a numerical solution.

We may be able to substitute it back into the equation.

Suppose a candidate proposes a SQL query.

We may be able to execute it in an isolated database and inspect the result.

Now selection becomes:

candidate
external check
pass / fail / score

This is qualitatively different from asking another model which answer sounds better.

The stronger the external signal, the more useful additional sampling can become.

Because the system can explore more alternatives while keeping the final decision tied to evidence.


13. The evaluated object is now the whole inference system

Suppose two experiments use the same base model.

Experiment A:

one sample
argmax / greedy decoding

Experiment B:

32 samples
high-temperature generation
verifier scoring
best candidate selected

Are they evaluating the same thing?

At the parameter level, yes.

At the system level, no.

The evaluated objects are:

A = model + decoding protocol

B = model + sampling budget + decoding protocol + verifier + selection rule

Reporting only the model name hides most of the computation that produced the result.

This is why inference-time scaling should be treated as architecture at the system level, even when the neural network weights do not change.


14. Compute budgets need units

Saying:

We used more test-time compute.

is incomplete.

More of what?

Possible budgets include:

generated tokens
number of samples
number of recurrent steps
number of verifier calls
number of search nodes
wall-clock latency
FLOPs
GPU-seconds

Two systems can spend the same number of model forward passes very differently.

For example:

System A
1 trajectory × 8,000 tokens

System B
8 trajectories × 1,000 tokens

Both generate roughly 8,000 tokens.

But their search structure is completely different.

A first-principles comparison should expose that difference.


15. Longer is not automatically better

The same caution applies to sequential reasoning.

If a model has already reached a useful state, more steps can introduce drift.

Tiny can exhibit this directly.

Suppose the score trajectory is:

step 1   0.55
step 2   0.68
step 3   0.81
step 4   0.84
step 5   0.80
step 6   0.71

The sixth step is not more intelligent merely because it came later.

This is why we recorded prediction trajectories rather than assuming recurrence helped.

The same principle holds for token-based deliberation:

Additional computation is a resource, not a guarantee.

It needs a stopping rule, selection rule, or evaluation signal that justifies its cost.


16. Evaluate before scaling

Before increasing inference compute, establish a baseline.

At minimum record:

single-sample accuracy
single-sample latency
single-sample token cost
failure categories

Then change one thing.

For example:

N = 1
N = 2
N = 4
N = 8
N = 16

Measure:

quality
latency
tokens
verifier calls
selection errors

The shape of the curve matters more than one final number.

You may discover:

1 → 4 samples     large gain
4 → 8 samples     small gain
8 → 16 samples    no gain

That tells you where additional compute stops earning its place.


17. A simple scaling experiment harness

We can make the experiment structure explicit:

import time


def evaluate_budget(
    prompts,
    solve,
    is_correct,
    budgets=(1, 2, 4, 8),
):
    rows = []

    for budget in budgets:
        correct = 0
        start = time.perf_counter()

        for prompt, target in prompts:
            answer = solve(prompt, budget=budget)
            correct += int(is_correct(answer, target))

        elapsed = time.perf_counter() - start

        rows.append({
            "budget": budget,
            "accuracy": correct / len(prompts),
            "seconds": elapsed,
        })

    return rows

The solve function may implement:

  • one long trajectory;
  • self-consistency;
  • Best-of-N;
  • verifier-guided search;
  • a recursive latent model.

The harness does not care.

That is useful.

It lets us compare inference strategies through a common contract.


18. Training changes a different thing

Inference-time scaling keeps the weights fixed.

Training changes the parameters.

At a high level:

training examples
objective
loss / reward signal
gradients or policy update
new parameters

This can alter the model so that behaviours requiring expensive inference procedures become more likely under cheaper inference.

But we should separate several mechanisms.

Supervised fine-tuning

input → desired output

Preference optimization

input + preferred / rejected outputs

Reinforcement learning

sampled behaviour
reward
policy update

Distillation

teacher behaviour
training data
student model

They are all post-training mechanisms.

They do not perform the same computation.


19. Preference learning was already one form of post-training signal

The previous chapter gave us:

A preferred to B

That is already enough to train a comparative model.

In larger language-model systems, preference information can also be used to alter the generator itself.

The important first-principles idea is not a particular algorithm.

It is that supervision can be relative rather than absolute.

That connects directly to our earlier distinction:

absolute score
      vs
relative preference

and to inference-time selection:

generate alternatives
compare alternatives
retain preferred result

Training and inference can therefore reuse the same conceptual signal in different places.


20. Reinforcement learning adds sampled outcomes and reward

A simplified reinforcement-learning loop is:

current policy
sample behaviour
measure reward
estimate which behaviour was better or worse than expected
update policy

For reasoning-oriented language models, the reward can sometimes be derived from a verifier.

For example:

math answer correct?        → reward
unit tests pass?            → reward
output format valid?        → reward component

This changes the training signal from:

imitate this exact target

into something closer to:

produce behaviour that earns this outcome

That is a major difference.

But again, the label RL is not the explanation.

We still need to ask:

  • what was sampled?
  • what produced the reward?
  • how sparse is the reward?
  • how is advantage estimated?
  • what update is applied?
  • what prevents destructive policy changes?

The acronym disappears when we open the loop.


21. Verifiable reward is powerful because it moves judgment outside the model

Suppose the task is arithmetic.

A generated trajectory may be long and persuasive.

But the final answer can often be checked mechanically.

That gives training an external signal:

model output
verifier
reward

The model does not have to grade its own reasoning.

This is important because self-evaluation and correctness are not the same capability.

A system can generate a flawed trajectory and confidently approve it.

An external verifier gives us a different source of information.

This is one reason reasoning systems become much easier to study when the domain provides strong checks.


22. Distillation moves computation across time

Distillation gives us another powerful way to think about compute.

Imagine an expensive system:

large model
+ many samples
+ verifier
+ selection

It produces high-quality trajectories.

We can use those trajectories as training data for a smaller or cheaper model:

expensive teacher system
     generated data
      student model
    cheaper inference

The expensive computation has not disappeared.

Some of it moved earlier in the lifecycle.

Instead of paying the full cost for every future request, we pay a larger cost while constructing the training set and then attempt to compress useful behaviour into the student’s parameters.

This gives us another key distinction:

runtime compute
      vs
training-time compute

Distillation is partly a strategy for moving information from one side of that boundary to the other.


23. This gives us five places to pay

We can now summarize the main levers in this book:

1. PARAMETERS
   larger or different architecture

2. INTERNAL COMPUTATION
   recurrence / iterative latent updates

3. INFERENCE SAMPLING
   more trajectories / search / voting

4. SELECTION AND VERIFICATION
   rankers / critics / tests / reward models

5. TRAINING
   supervision / RL / distillation / optimization

These are not mutually exclusive.

A modern system may use all five.

But separating them gives us a diagnostic language.

Instead of saying:

The reasoning model is stronger.

we can ask:

Did the base parameters improve?
Did the system run for more steps?
Did it sample more candidates?
Did the selector improve?
Did the verifier improve?
Did post-training improve the policy?
Did distillation compress expensive behaviour?

That is a much more useful analysis.


24. The same model can produce several different systems

Suppose we freeze one language model completely.

We can still construct:

System A — single sample

prompt
model
answer

System B — self-consistency

prompt
model × N
answer extraction
majority vote

System C — Best-of-N

prompt
model × N
ranker / verifier
best candidate

System D — iterative refinement

prompt
draft
critique
revision
comparison

Same base parameters.

Different runtime systems.

If their benchmark scores differ, it would be misleading to attribute the entire difference to the base model.


25. Search introduces partial trajectories

So far our multi-sample examples generated complete candidates.

Another possibility is to branch before completion.

Conceptually:

                state 0
               /      \
          state 1A   state 1B
           /   \       /   \
        2A     2B    2C    2D

Now the system can score or prune partial trajectories before paying to complete every branch.

This is different from Best-of-N.

Best-of-N is approximately:

generate complete leaves
score leaves

Search over partial trajectories is:

generate prefixes
score / prune
expand survivors

The computation budget may be similar.

The allocation strategy is not.

This is why the phrase test-time compute hides important structure.


26. Search only helps if partial states are scoreable

Branching introduces another problem.

How do we know which unfinished trajectory is promising?

A final-answer verifier may not help yet.

We may need:

  • a value model;
  • a heuristic;
  • a learned critic;
  • a partial-state verifier;
  • a domain-specific bound.

Now the architecture begins to reconnect with EBT and SICQL.

Remember V:

How good is the current state?

That question becomes extremely useful when we must decide whether a partial trajectory deserves more compute.

The chapters are not isolated after all.

A value head that looked abstract earlier now has a concrete systems role:

partial state
V estimate
continue / prune / prioritize

27. Reasoning performance is a property of a pipeline

A more complete picture now looks like this:

input
base model / policy
trajectory generation
optional branching
selection / verification
final result

Training sits underneath:

observed trajectories
labels / preferences / rewards
optimization
new model parameters

Distillation can feed system outputs back into training data:

expensive inference system
      trajectories
      student training

Once we see the full pipeline, the phrase reasoning model becomes less mysterious.

There may be a model in the center.

But the observed capability belongs to the complete process.


28. Visible chain-of-thought is not our definition of reasoning

It is tempting to define a reasoning model as one that prints intermediate reasoning text.

That definition is too narrow for this book.

Tiny performs iterative latent computation without producing a natural-language chain.

A search procedure can reason over explicit partial states.

A verifier can reject a candidate without generating any explanation.

A distilled model may produce a short answer even if the teacher system used much more computation to create its training data.

So our working definition should focus on computation rather than presentation:

Reasoning-oriented systems allocate structured computation to transform, explore, compare, or verify intermediate states before committing to an answer.

Visible reasoning text can be one representation of that process.

It is not the process itself.


29. Failure mode: more samples, same mistake

If all samples are highly correlated, increasing N may buy very little.

Symptoms:

candidate wording changes
core mistake remains identical

Measure diversity at the level that matters.

For a math problem, textual diversity is less important than solution-path diversity.

For code, different formatting is less important than genuinely different implementations.

Do not confuse surface variation with search coverage.


30. Failure mode: the verifier can be gamed

Suppose the selector rewards a particular format strongly.

The generator may discover outputs that satisfy the format while missing the real task.

Or a reward model may prefer confident style over correctness.

Then more inference-time search can optimize the wrong objective more effectively.

This is not a failure of search itself.

It is a failure of the evaluation signal.

The system is doing exactly what its scoring mechanism asked it to do.


31. Failure mode: training and inference optimize different things

Suppose training rewards exact correctness.

But inference chooses candidates using a style-oriented preference ranker.

Now we have two objectives:

training objective     correctness
inference selector     style preference

The final system may move away from the capability training created.

Every selection stage should therefore be treated as another objective in the pipeline.


32. Failure mode: distillation copies artifacts

Teacher-generated data can contain:

  • systematic errors;
  • verbosity habits;
  • formatting quirks;
  • spurious shortcuts;
  • domain blind spots.

Distillation can compress those too.

A larger teacher is not automatically a source of ground truth.

Generated data should still be filtered, verified, or sampled under a protocol appropriate to the task.


33. Failure mode: comparing unequal systems

Suppose we compare:

Model A
single sample

Model B
64 samples + verifier

and conclude:

Model B has the better architecture.

That conclusion does not follow.

The experiment changed both model and inference system.

A fair comparison might include:

A single
A scaled
B single
B scaled

Now we can separate architecture effects from inference effects.

This is the same experimental discipline we used when comparing optimizers.

Change one axis at a time when possible.


34. Build an experiment matrix

A useful model-study matrix is:

Model Steps / samples Selector Training Metric
Tiny 1 step none BCE accuracy
Tiny 6 steps none BCE accuracy
LLM A 1 sample none base task score
LLM A 8 samples majority base task score
LLM A 8 samples verifier base task score
LLM A 8 samples verifier RL task score
Student 1 sample none distillation task score

Now the comparison has axes.

We can ask:

What did recurrence add?
What did sampling add?
What did the verifier add?
What did RL add?
What did distillation retain?

That is much more informative than one leaderboard column.


35. Record cost beside quality

Every row should also record cost.

At minimum:

quality
latency
tokens or model calls
memory

Because a system that improves accuracy from:

80% → 81%

while increasing inference cost:

1× → 64×

may be a poor engineering trade-off.

Another system may gain less absolute quality but dominate under the actual deployment budget.

The right model is often a point on a cost-quality frontier, not the single highest number.


36. Where should extra compute live?

We can now ask a more mature design question.

Suppose the current system is not good enough.

Where should we spend the next unit of complexity?

Add parameters when

The model lacks representational capacity and additional architecture is justified by evidence.

Add recurrent steps when

Intermediate state improves across steps and the same learned mechanism benefits from additional computation.

Add samples when

The model can generate genuinely different useful candidates and a reduction rule can exploit that diversity.

Add a selector or verifier when

Generation is stronger than recognition, or when external evidence can distinguish candidates more reliably than the generator itself.

Change training when

The desired behaviour is not sufficiently represented in the model’s current parameters.

Distill when

An expensive teacher or inference system produces behaviour worth compressing into a cheaper deployment model.

This is not a formula.

It is a decision checklist.


37. The first-principles rule still holds

At the beginning of the book, we said:

A complicated model becomes understandable when we recursively decompose it.

Now the object being decomposed is larger.

Not just:

model

but:

reasoning system

Open it:

reasoning system
base model
inference protocol
sampling budget
state / trajectory structure
selector
verifier
training objective
optimizer
distillation process

Open each of those again.

Eventually we return to ordinary operations:

matrix multiplication
softmax
sampling
loops
comparisons
losses
moving averages
parameter updates

The mystery disappears in exactly the same way.


38. What this changes about the models we already built

MR.Q is not merely a scoring model.

It can become a Best-of-N selector.

EBT is not merely a multi-head network.

Its V head suggests a mechanism for valuing partial states.

SICQL is not merely model composition.

It shows how independent decision components can support a larger inference system.

HRM is not simply a more sophisticated network.

It is one way to allocate additional sequential computation.

Tiny is not proof that recurrence equals reasoning.

It is a controllable experiment in repeated latent computation.

PACS is not a reasoning mechanism.

It changes how all these models learn.

Preference Rankers are not only training models.

They can also become runtime selectors.

The models now fit into a larger map.


39. What we have learned

Reasoning performance can improve without changing the base architecture.

It can improve because we:

run longer
sample wider
search partial trajectories
select better
verify externally
train differently
distill expensive behaviour

Those mechanisms are not interchangeable.

They represent different locations for computation and different assumptions about where the system’s current limitation lies.

The central distinction is:

architecture
inference strategy
selection
training

A complete reasoning system may contain all four.


40. The question to carry into model selection

We are now ready for the final comparison chapter.

Earlier, model selection looked like a question about architecture:

MR.Q, SICQL, HRM, or Tiny?

That is no longer enough.

The better question is:

What decision must this system make, where is its current limitation, and where should the next unit of computation live?

Perhaps the answer is a larger network.

Perhaps it is one more recurrent step.

Perhaps it is eight independent samples.

Perhaps it is a verifier.

Perhaps it is better supervision.

Perhaps it is a distilled student.

And sometimes the right answer is simpler:

do not add anything.

If the current model already solves the decision reliably, more machinery is just more machinery.

That brings us back to the engineering rule that has followed the entire book:

Use the simplest mechanism that solves the decision you actually have.


Further reading

The immediate motivation for this chapter came from Sebastian Raschka’s Build a Reasoning Model (From Scratch) material, which presents a progression from evaluation to inference-time scaling, reinforcement learning, and distillation on top of a pretrained language model:

For the specific mechanisms discussed here:

These references use much larger language models than the small PyTorch systems in this book. The reason to include them is not to copy their scale. It is to expose the same engineering question at a larger level:

Where does the useful computation live?