Part 1 — Where You Stand

Seven operations, one of them intelligent

Take the paragraph review from Chapter 1 and write down everything the process actually has to do. Not the prompt — the process.

1. Select the paragraph to review.
2. Decide whether it has changed since the last review.
3. Decide whether there is budget left for a call.
4. Identify claims in the paragraph that need supporting evidence.
5. Extract the flagged sentence and its position.
6. Check that a cited source resolves and that its year matches the bibliography.
7. Decide whether to write the change to the file.

Six of those have exactly one correct answer, computable without a model. Selection is an index lookup and change detection is a hash comparison. Budget is arithmetic; extraction is parsing. Source checking resolves the request and compares it against the bibliography. The write decision evaluates policy.

One of them — step 4 — requires reading a sentence and forming a judgment about what would count as support for it. That is the operation you cannot specify well enough to implement.

Now look at how this is usually built. One prompt: “Review this paragraph, check the citations, and tell me if it’s ready to publish.” Seven operations, all seven routed through the stochastic component, and now the budget arithmetic can be wrong.

Which parts of a process are permitted to be unreliable, and who decides?

The rule

A component belongs on the deterministic side unless it demonstrably cannot be. If you are in doubt about which side it belongs on, that doubt is itself the evidence: it belongs on the deterministic side.

The second sentence sounds like a rhetorical flourish. It is not; it is an argument, and it is worth making explicit because it is the load-bearing claim of this book.

To be in doubt about whether an operation needs a model, you must have some idea of what the correct answer would look like. You are weighing “could I just compute this?” against “is this too fuzzy?” — and to weigh that at all, you must be holding a rough specification in your head. That mental object is the thing you were claiming not to have.

The operations that genuinely require a model are not the ones that feel hard. They are the ones where you cannot state the acceptance criterion at all — where you would know a good answer when you saw it, and two competent people would produce different answers, both acceptable. There is no doubt in those cases. You are not wondering whether to write a rule for “identify claims that need evidence.” You already know you cannot.

So the operational test is not is this hard? It is:

Can you write down what a correct answer would be, without writing down the answer?

If you can, you have a specification, and a specification is an implementation you have not typed yet.

The asymmetry that makes any of this possible

That test has a second edge, and it is the more valuable one.

Very often you cannot write the solution but you can write the check. You cannot write a function that repairs a broken parser, but you can run the test suite. You cannot write a function that produces a good paragraph review, but you can check that every sentence the review flagged actually exists in the paragraph, that every source it cites resolves, and that no source postdates the claim.

Generation is hard; verification is cheap. This asymmetry — familiar in complexity theory, and used here as an analogy rather than a formal claim — is the reason applied AI is possible at all. If checking an answer were as expensive as producing one, no amount of process engineering would help you; you would simply have two unreliable components instead of one.

Programs make the same division concrete. PAL has a language model write intermediate programs and hands them to an interpreter to execute, so the model proposes and exact machinery computes (Gao et al., 2023). Chapter 4 looks at benchmarks built deliberately on this asymmetry. The bound travels with it: the asymmetry holds where someone has manufactured a cheap check. Where verification is delayed, subjective, or adversarial — fraud detection, in Chapter 4 — it does not.

So the split is not “model versus code.” It is:

    flowchart LR
    subgraph DET1["deterministic"]
        direction TB
        A["select · budget · policy"]
        B["context assembly<br/>explicit, hashed"]
    end
    subgraph STO["stochastic — confined"]
        M["model call<br/><i>proposes</i>"]
    end
    subgraph DET2["deterministic"]
        direction TB
        P["parse constrained output"]
        V["verify<br/><i>tests · source checks · reproduction</i>"]
        D["decide · act · record"]
    end
    A --> B --> M --> P --> V --> D
    D -.->|"unverified: retry, escalate, or stop"| A
  

One stochastic box. Deterministic on both sides. Proposals flow out of the stochastic core; only verified things flow back into the process state.

Its unreliability is the product

Here is where the rule stops being a counsel of caution and becomes a design principle.

A deterministic function can only return an answer from the set you enumerated when you wrote it. That is exactly what makes it reliable, and exactly what makes it useless for the one operation you needed help with. If you already knew the space of acceptable answers, you would not be reaching for a model.

What you are buying from a stochastic component is coverage of a space you did not enumerate. The model proposes readings you did not anticipate. Sometimes that proposal is wrong; sometimes it is the thing you paid for. Those are the same property, observed from two sides. You cannot purchase the surprise and decline the variance.

This yields a blunt and surprisingly practical test:

If you would be annoyed to get a different answer when you re-run it, it should not be a model call.

Annoyance at variance means you wanted a fixed answer. Wanting a fixed answer means you could describe the fixed answer. Which means, per the rule, you should have implemented it — and instead you are paying money and latency for variance you do not want, in a component whose failures are harder to localize than a function’s.

Turn the test around and it tells you where a model genuinely earns its place. You want three different reviewers to flag three different things. You want several independent repair attempts to fail in uncorrelated ways, because then one of them may succeed where the others did not. Variance is the mechanism. Part 5 is entirely about that: whether independent, differently-seeded, differently-framed calls actually buy coverage, measured rather than assumed.

You cannot configure your way onto the deterministic side

The obvious objection: fine, but I’ll set temperature=0 and treat the call as deterministic.

That option is not available, and the reason is more interesting than the usual “floating point is fuzzy” hand-wave.

Horace He and colleagues at Thinking Machines Lab sampled 1,000 completions at temperature 0 from Qwen3-235B-A22B-Instruct-2507, same prompt, 1,000 tokens each. They got 80 unique completions. The most frequent one appeared 78 times. Every completion was identical for the first 102 tokens; divergence began at token 103, where 992 completions said “Queens, New York” and 8 said “New York City” (He et al., 2025).

The usual explanation — floating-point non-associativity plus nondeterministic GPU scheduling — turns out to be the wrong diagnosis. The kernels involved are run-to-run deterministic. The actual culprit is that reduction kernels are not batch-invariant: the arithmetic reduction order depends on the batch size, and the batch size depends on server load. Make the kernels batch-invariant and all 1,000 completions come back identical.

Sit with the consequence. Under standard serving, your temperature-0 output is a function of how many other people were hitting the endpoint at the same moment. Nondeterminism is arriving from outside your process, through a variable you cannot see, do not control, and could not log.

It gets worse for anyone hoping to pin this down by fixing their own environment. Yuan and colleagues showed that changing evaluation batch size, GPU count, or GPU version alters generated responses under greedy decoding, and that reasoning models amplify it, because a rounding difference in an early token cascades into a different chain of thought. Under bfloat16 with greedy decoding, DeepSeek-R1-Distill-Qwen-7B showed up to 9% variation in accuracy and 9,000 tokens of difference in response length attributable to GPU count, GPU type, and batch size alone (Yuan et al., 2025). Their mitigation, LayerCast, keeps weights in 16-bit but performs computation in FP32 — a change to the inference stack, not to an API parameter.

Bound these results properly. Both are studies of open-weight models on controlled inference stacks; neither measured a commercial hosted endpoint, and He and colleagues’ fix requires control over the kernels. But the direction of the finding is what matters architecturally, and it runs one way: greedy decoding removes the sampler, not the nondeterminism.

The consequence for evaluation follows immediately, and Song and colleagues make it directly: evaluating a model on one output per example hides real performance variability (Song et al., 2024). One run is a sample, not a measurement — which is why the experiments in Part 5 report repeated draws, and why Chapter 27 spends an entire chapter on a replication in which a promising signal did not survive.

So determinism is not a flag. You do not move a component to the deterministic column by configuring it. You move it by replacing it with something that computes the answer.

Classifying an operation

The decision procedure, applied to the seven steps we started with:

That last row is the one that catches real systems. Parsing a model’s prose answer into structured fields is a deterministic operation, and reaching for a second model call to do it is the most common instance of the mistake this chapter is about.

The three deterministic rungs below reuse the paragraph review’s own steps. Each runs exact and repeatable:

import hashlib

prev_sha = hashlib.sha256(paragraph.encode()).hexdigest()
changed = hashlib.sha256(paragraph.encode()).hexdigest() != prev_sha

budget_left = budget_total - calls_made * cost_per_call
may_call = budget_left >= cost_per_call

span = paragraph.find(flagged_sentence)  # -1 means paraphrased: escalate, don't re-prompt
if span == -1:
    decision = "escalate"

Run it and two failure states become visible without any model: changed goes true the moment an edit lands after the review (the stale-review bug from Chapter 1), and span == -1 fires when the review paraphrases instead of quoting, which is exactly where teams are tempted to add a second model call to parse the first one’s prose. The reader can now do one new thing: replace three model-routed operations with code and keep the failure observable.

The book eventually measures a small instance of the rule. In Chapter 28, ten of forty extraction items are already structured, lines like CDN_TTL_SECONDS=86400. A deterministic rule resolves all ten with no model call. The strong model, sent the same items, gets nine: on one it answers with the bare digits and drops the key, and the check can no longer tie the number to what it measures. Where the operation was already specified, the model added a way to fail and nothing the rule lacked.

Where the rule breaks

A rule stated this firmly needs its own failure mode, and this one has a sharp one.

Determinism is cheap when the specification is cheap. When the specification is the dominant cost — open-ended natural language, a long tail of formats, judgments that resist enumeration — a hand-built rule cascade is not reliability. It is four hundred lines of brittle regex that someone maintains forever, fails silently on input twelve months from now, and is less trustworthy than a model call with a verifier behind it. “If in doubt, deterministic” does not license building a bad expert system in 2026.

When a deterministic step starts getting hard, check this first:

When a deterministic step becomes difficult, the usual cause is an unconstrained upstream interface, not a genuine need for intelligence.

Extracting a flagged sentence from free-form model prose is genuinely miserable. That is not evidence that extraction needs a model. It is evidence that the generation step should have been constrained to emit a schema. Fix the interface, and the deterministic step becomes trivial again. Add a second model call to parse the first one’s output, and you have doubled your stochastic surface to avoid writing a schema.

The honest boundary, then: the rule is a strong default with a clear override. Override it when the specification cost genuinely exceeds the value of determinism — and record that you did, because that is a design decision someone will need to revisit.

What this buys you in money

Cost is not a side note in this book; it is a structural constraint, and it points the same way.

Every operation on the deterministic side is free and repeatable. Every operation on the stochastic side costs tokens and latency against a finite budget. The stochastic budget is the thing you are actually purchasing, and it is the only place variance can earn its keep — in Part 5, on independent draws, where extra sampling might buy coverage.

Spending that budget on budget arithmetic, sentence extraction, or change detection is not merely inelegant. It is spending the scarce resource on the operations that needed it least, which leaves less for the operations that needed it most. The determinism default is the cost-control mechanism. This is why the runtime we build records usage and cost for every attempt (Chapters 11 and 13) and why its scheduler treats an exhausted budget as a first-class stopping condition (Chapter 28).

Do this now

Twenty minutes. Classify a real pipeline.

1. Take one workflow you already run with AI — yours, or one at work. Write down every operation it performs, the way this chapter listed seven for the paragraph review. Aim for at least six.
2. For each, run the test: can you write down what a correct answer would be, without writing down the answer? Mark it D (deterministic) or S (stochastic).
3. Apply the annoyance test to every S: would you be irritated to get a different answer on a re-run? Every yes is misclassified. Move it.
4. Count how many operations currently go through a model, and how many actually need to.

The gap between those two numbers is your entire cost-reduction program, and Chapter 6 puts a price on it.

Failure modes

• Routing exact operations through the model. Asking for a token count, a date comparison, or a budget decision from a component that samples.
• Treating temperature=0 as determinism. Measured false (He et al., 2025; Yuan et al., 2025). It removes the sampler and leaves the batch dependence.
• A second model call to parse the first one’s output. Doubles the stochastic surface to avoid constraining an interface.
• Averaging away disagreement. When two competent judgments differ, the spread is data. Collapsing it to a mean discards what the stochastic component actually produced.
• Over-applying the rule. A rule cascade whose specification cost exceeds the value of determinism, maintained forever, failing silently.
• Single-run belief. One passing result is a draw, not an estimate (Song et al., 2024).

What this chapter established

• The governing rule: deterministic unless demonstrably impossible, with doubt counting as evidence for determinism — because doubt requires a specification, and a specification is an untyped implementation.
• The operational test: can you state what a correct answer would be, without stating the answer?
• The productive asymmetry: where you cannot write the solution but can write the check, the architecture is a stochastic generator behind a deterministic verifier.
• Stochasticity is not a defect to minimize to zero. It is the mechanism that produces proposals you could not enumerate, and it should be confined rather than eliminated.
• temperature=0 does not deliver determinism. Under standard serving, output depends on concurrent load through batch-size-dependent reduction kernels; 1,000 identical greedy requests produced 80 distinct completions.
• The override condition: when specification cost dominates, use a model — and write down that you chose to.

Next

We now have a rule for which side of the line an operation belongs on, and an architecture for the cases where you cannot write the solution but can write the check: a stochastic generator behind a deterministic verifier.

That architecture is not hypothetical, and the next chapter goes looking for it in the wild. It turns out that one industry spent twenty years building exactly that verifier — compilers, type systems, test suites, continuous integration — for reasons that had nothing to do with AI, while simultaneously producing millions of written specifications and filing them under project management. That is why software automated itself first, and it is not because code is easy. The chapter turns the observation into a survey you can run on any domain.

Continue with Scrum Built the Training Set.

References

• Horace He and Thinking Machines Lab. Defeating Nondeterminism in LLM Inference. Thinking Machines Lab: Connectionism, September 10, 2025. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
• Jiayi Yuan, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference. arXiv:2506.09501, 2025. https://arxiv.org/abs/2506.09501
• Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism. arXiv:2407.10457, 2024. https://arxiv.org/abs/2407.10457
• Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: Program-aided Language Models. ICML, PMLR 202:10764–10799, 2023. https://proceedings.mlr.press/v202/gao23f.html