Part 1 — Where You Stand

The thing the marketing does not cover

Every vendor will sell you more capability. None of them can sell you direction.

A more capable model pointed at the wrong objective does not produce less wrong work. It produces better-executed wrong work at greater volume, wrapped in more convincing prose. The gap between what you wanted and what you got does not shrink with capability. Frequently it widens, because a weaker model’s output was obviously off and a stronger model’s output is plausibly off.

Capability is a scalar. What you need is a vector.

Chapter 6 ended on a purchasing question — what are you buying and does it ever stop. This chapter is the prior question, and it is the one that determines whether any of that spending was worth it: how would you know?

How do you measure whether a stochastic process is getting better at something you cannot define precisely?

What Part 1 has actually been about

It is worth noticing that the last four chapters have been circling one problem from four directions.

Chapter 4 said that without a cheap verifier you have a demo rather than a process. Chapter 5 said you cannot tell when you have become a rubber stamp. Chapter 6 said you cannot tell whether the expensive model earned its price.

Those are not three problems. They are one problem wearing three coats: you cannot see the quality of what you are producing. Verification, review discipline, and cost control all bottom out in measurement, and if measurement is absent, all three degrade silently and simultaneously.

That is why this is the last chapter of Part 1 rather than a technical appendix. Everything before it assumed you would be able to tell whether it worked.

Why the deterministic playbook does not transfer

You already know how to measure software. Uptime, latency percentiles, error rate, throughput, test pass rate. These work extraordinarily well, and almost none of them transfer.

That last row is where the real work is, and it is why this feels harder than it should. A 99.9% uptime target is measurable because down is defined. “The review was good” is not defined by anyone, anywhere, until you define it. The act of definition is not preparation for the measurement. It is the measurement, and it is the part people skip.

The second row deserves its own warning. Deterministic systems announce their failures; that is the entire basis of alerting. A stochastic process degrading from 94% to 88% acceptable output emits no exception, triggers no page, and looks exactly like a system working fine. You find out from a customer, months later — which is Chapter 5’s unbounded blast radius arriving through a different door.

Evals are experiments

The most useful reframe available here is Evan Miller’s: evaluations are experiments, and the literature on evaluation has largely ignored a century of work on how to analyze and plan experiments (Miller, 2024). Treating an eval as a number you print, rather than as an estimate with uncertainty, is the root of most bad decisions in this area.

His recommendations are concrete, and they are the methodology this book uses:

1. Report standard errors on eval scores, computed from the Central Limit Theorem, treating the questions as drawn from an unseen super-population.
2. Use clustered standard errors when questions come in related groups — several questions about one document, several tasks from one repository — because those questions are not independent draws.
3. Reduce variance by resampling. Running each question K times reduces the conditional variance proportionally. Where you have access to next-token probabilities, using them removes sampling variance entirely.
4. When comparing two models, do inference on question-level paired differences, not on the two population-level summary statistics. The models are correlated across questions, and exploiting that correlation shrinks the standard error substantially.
5. Run a power analysis to find out whether your eval is even capable of detecting the effect you care about.

Number five is the one that should change behavior this week. The standard industry practice is to compare two models on twenty examples, observe that one won fourteen to six, and adopt it. Two equally good models split at least that lopsidedly about one time in nine, and a power analysis would have shown in advance that twenty examples cannot reliably detect the kind of difference usually being claimed. An enormous amount of confident model-selection folklore is built on evals that had no power to support it.

The arithmetic fits in a few lines, and it is worth running once by hand:

from math import comb, sqrt

# Two models, 20 head-to-head items, one wins 14. If they were equally good,
# how often would a split at least this lopsided happen?
n, wins = 20, 14
p_two_sided = 2 * sum(comb(n, k) for k in range(wins, n + 1)) / 2**n   # ≈ 0.115

# A pass rate is an estimate, not a fact: 40 passes out of 50 runs.
rate, runs = 40 / 50, 50
standard_error = sqrt(rate * (1 - rate) / runs)                          # ≈ 0.057

A pass rate of 80% with a standard error near six points means “somewhere around 69% to 91%”. Two models whose intervals overlap that much have not been told apart. (The fourteen-six calculation assumes independent items; paired, clustered designs, which Miller recommends, do better, and later chapters use them.)

And number three is the direct consequence of Chapter 3. If temperature=0 does not give you determinism, then a single run is a draw from a distribution, and reporting it as a measurement is a category error.

The ladder of what you can measure

Work down this list in order and do not skip rows, because the cheap rows are free and the expensive rows are where people start.

Two things about this ordering.

Most teams start at level 5 because it is the most exciting and go no further. Level 1 costs nothing, catches a surprising share of real failures, and — per Chapter 3 — is the row you should exhaust before any of the others. Level 3 is nearly free and is the only row that is hard for the system to game, because it is downstream of somebody actually using the thing.

And note what level 4 tells you about level 5’s ceiling, which is the next section.

The critic model, engineered properly

A model that scores your outputs, independent of the model doing the work, evaluated separately from it — that is the right instrument, and it is the only one that scales to continuous measurement. It is also easy to build badly in ways that produce numbers which look like measurements and are not.

Five constraints, each with a reason.

It must be independent of the generator. Not merely a separate call — a different model family. LLM judges favor their own outputs, and the strength of that preference scales linearly with the model’s ability to recognize its own text (Panickssery et al., 2024). Better still, use a panel: a jury of smaller models from disjoint families beat a single large judge with less intra-model bias at over seven times lower cost (Verga et al., 2024). A different family removes self-preference; it does not guarantee different mistakes. Whether it does is something to measure, and Part 5 measures it.

It must be validated against ground truth, and that validation must travel with every number it produces. Build a small gold set — a few hundred items labeled by people — and measure the critic’s agreement with it. Then report that agreement figure alongside every result the critic ever gives you. A judge whose agreement you have not measured is not an instrument; it is another generator with opinions. PaperBench shows the industrial form of this discipline: replicating 20 research papers is graded against rubrics broken into 8,316 individually gradable tasks, co-developed with the papers’ own authors, and the automated judge is assessed on a separate benchmark of its own (Starace et al., 2025). Decompose the judgment until the authors of the ground truth would recognize it, then validate the judge separately.

The calibration target is not 100%, and this is the useful part. Zheng and colleagues found GPT-4 reached roughly 85% agreement with human preferences on open-ended questions — while humans agreed with each other only 81% of the time (Zheng et al., 2023). So:

You cannot exceed the reliability of your ground truth. An evaluation reporting near-perfect agreement is measuring something narrower than it claims, or has a gold set too easy to discriminate.

It must be frozen and versioned. This is the practical failure that ruins real measurement programs, and it gets almost no attention. You want a time series — is the process getting better over months? A critic that silently changes underneath you destroys that series. Providers update model snapshots. Endpoints get deprecated. A prompt tweak someone made on a Thursday shifts every score after it.

So pin the critic: model, version, prompt, parameters, all of it, recorded with each score. When you must change it, re-score the historical archive with the new critic to establish the offset before comparing across the boundary.

Which means you must have kept the archive. This is the point where Part 3 stops being administrative hygiene: re-scoring requires the preserved outputs (Chapters 11 and 17), and rebuilding the time series requires the log of what was scored, when, and by which critic (Chapters 16 and 18).

Never optimize against your only critic. Gao, Schulman, and Hilton studied this directly, using a fixed “gold” reward model to stand in for human judgment and a proxy reward model trained from its labels. As you optimize against the proxy — by RL or by best-of-n sampling — gold performance rises, then turns and degrades. The functional form differs by method, and the coefficients scale smoothly with reward model size (Gao et al., 2023).

That is Goodhart’s law with a measured shape. The practical consequence: keep a held-out critic you never tune against, and check the two periodically. When the optimized-against score keeps climbing and the held-out score stops, you have crossed the turn.

The critic must not see what would bias it. If the critic can tell which arm, which model, or which prompt variant produced an answer, it is not scoring the answer. Blind it — strip provenance, randomize order, normalize length. This is the same isolation discipline Part 5 applies to independent calls, applied to the thing doing the judging.

    flowchart LR
    G["generator<br/><i>the work</i>"] --> O["output<br/><i>preserved, blinded</i>"]
    O --> M1["1 · mechanical checks"]
    M1 --> C["critic panel<br/><i>pinned version</i>"]
    C --> S["score + critic id<br/>+ agreement figure"]
    GOLD["gold set<br/><i>human-labeled</i>"] -.->|"validates"| C
    HELD["held-out critic<br/><i>never optimized against</i>"] -.->|"detects the turn"| S
    style HELD stroke-dasharray: 4 4
  

Build this before you build anything else

Part 1 has been argument. Here is the part that is not, and it is deliberately the first thing you make.

Build the measurement before you build the feature. Not because it is virtuous, but because of a simple asymmetry: if the harness exists first, every subsequent change in this book is testable the day you make it. If it exists last, you have a year of undocumented decisions that the harness can no longer test.

The smallest honest version, which takes an afternoon:

Do this now.

1. Freeze 30–50 real tasks from your actual domain. Real inputs, not invented ones. Where a correct outcome exists, record it. For open-ended tasks, record the acceptance criteria.
2. Write the mechanical checks that apply — output parses, cited sources resolve, quoted spans appear verbatim in the source, numbers sum. Level 1 of the ladder, and free.
3. Record a baseline. Today’s model, today’s prompt, each task run at least 3 times. Report mean pass rate with a standard error, not a single number.
4. Write down the date, the model, the model version, and the prompt hash. Attach these to every score, or later comparisons drift.
5. Store it where a later process can read it — a directory of JSON is entirely sufficient today.

That artifact is the reference point for the entire rest of this book. Chapter 11 will add the instrumentation that feeds it. Part 5 is nothing but this harness, run seriously, across four preregistered experiments — including one where the result the author wanted did not replicate, which is exactly the outcome a harness exists to deliver.

If you build nothing else from Part 1, build this.

Where this gets hard

• Frozen sets rot. Your corpus stops representing production as production drifts, and the score stays flat while quality falls. Refresh on a schedule, and keep the old set alongside the new one so the comparison survives.
• Small corpora cannot detect small effects. Power analysis will tell you your eval is underpowered. The honest response is to stop claiming small wins — not to reach for a bigger model.
• The things that matter most are often unmeasurable. “Did this book get better” lies outside any frozen task set. For those, measure the process rather than pretending to measure the output: was it reviewed, against what criteria, by whom, with what evidence attached. A recorded process is a weaker claim than a measured outcome and an enormously stronger one than a feeling.
• Measurement costs money from the same budget as the work. Chapter 6’s arithmetic applies to the critic too, which is another argument for exhausting the free rows of the ladder first.
• Goodhart is not avoidable, only managed. Any metric you optimize becomes less informative about the thing it proxied. Held-out critics and downstream signals slow this; they do not stop it.
• A critic is a model, and everything in this book about models applies to it. It is stochastic, it has a jagged frontier, and it will be confidently wrong in its own characteristic ways.

Failure modes

• Buying capability without direction. More intelligence aimed wrongly is faster, more convincing wrong work.
• Reporting a single run as a measurement. One run is a draw. Report n and a standard error or report nothing.
• Comparing models on twenty examples. Almost certainly underpowered; the winner is frequently noise.
• Unpaired comparison. Comparing two summary statistics throws away the correlation that would have made the test sensitive.
• An unvalidated judge. A critic with no measured agreement is an opinion generator with a numeric output format.
• An unpinned judge. Silent critic drift destroys the time series you were building, retroactively.
• Optimizing against your only critic. Proxy score rises while true quality turns; measured, with a known shape.
• Letting the generator’s family judge its own output. Self-preference scales with self-recognition.
• Starting at level 5 of the ladder. Mechanical checks and downstream signals are free and were skipped.
• Waiting until after the build to measure. Then nothing you did in between is evaluable.

What this chapter established

• Capability is a scalar and direction is a vector; vendors sell the former and only measurement supplies the latter.
• Verification, review discipline, and cost control are one problem — you cannot see the quality of what you produce — which is why measurement is the condition everything in Part 1 depends on.
• The deterministic playbook does not transfer: one run is a sample, failure is plausible rather than loud, quality is per-output, the metric is gameable, and “good” must be defined by you before anything can be measured.
• Evals are experiments. Report standard errors; cluster them when questions come in groups; resample to cut variance; compare models on paired question-level differences; run a power analysis before believing a result.
• The measurement ladder runs mechanical checks → frozen task set → downstream signal → human pairwise → critic model. Exhaust the free rows first.
• A critic must be independent of the generator, validated against a gold set with its agreement reported alongside every number, pinned and versioned, blinded to provenance, and never the sole optimization target.
• The ceiling is your ground truth: GPT-4 reached ~85% agreement with humans, who agreed with each other 81% of the time.
• Optimizing against a proxy improves true performance and then measurably degrades it. Keep a held-out critic to detect the turn.
• Build the harness before the feature. Thirty to fifty frozen tasks, mechanical checks, a baseline with error bars, and a recorded model/prompt identity.

Next

There is one more question, and it is the one this whole part has been building toward without saying so.

If the capability is real — and it is — and if you measure properly, then a straightforward prediction follows: things should be finishing. Backlogs should burn down. Projects should reach the terminal state where there is genuinely nothing left to do, sooner and more often. A tireless colleague working at a hundred times your rate should close a ten-year project in about five weeks.

That is not happening, or at least nobody has instrumented the thing that would show it. The last chapter of Part 1 asks why, and the answer turns out to be arithmetic rather than opinion — and it explains why the remaining five parts of this book are shaped the way they are.

Continue with Where Are the Finished Projects?.

References

• Evan Miller. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640, Anthropic, 2024. https://arxiv.org/abs/2411.00640
• Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36 (NeurIPS) Datasets and Benchmarks Track, 2023. https://arxiv.org/abs/2306.05685
• Leo Gao, John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization. Proceedings of the 40th International Conference on Machine Learning (ICML), PMLR 202, 2023, pp. 10835–10866. https://proceedings.mlr.press/v202/gao23h.html
• Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models. arXiv:2404.18796, 2024. https://arxiv.org/abs/2404.18796
• Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. PaperBench: Evaluating AI’s Ability to Replicate AI Research. arXiv:2504.01848, 2025. https://arxiv.org/abs/2504.01848
• Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM Evaluators Recognize and Favor Their Own Generations. Advances in Neural Information Processing Systems 37 (NeurIPS), 2024. https://arxiv.org/abs/2404.13076