← Inference From First Principles

Reasoning Models and Inference-Time Compute

A 0.6-billion-parameter model that can think, on 200 maths problems: accuracy against thinking budget, against parallel samples, and against an adaptive rule, with the tokens, the waiting time and the cost per problem solved; and the provider-side controls, versioned and untried.

Qwen3-0.6B was asked to solve 200 grade-school maths problems using five thinking budgets: 128, 256, 512, 1,024 and 1,536 generated thinking tokens. Accuracy was 53.5%, 52.0%, 54.5%, 49.0% and 48.5%. The 95% Wilson intervals for each accuracy span roughly seven percentage points on either side; the neighbouring-budget comparisons do not establish an improvement from longer thinking. A non-thinking answer scored 55.5%, and an eight-sample non-thinking vote scored 62.5%. The latter used about 2,114 generated tokens per problem rather than the 1,068 counted for a 1,024-token thinking budget, but it achieved more correct answers. Estimated mean waiting time was 20.4 seconds for a concurrently executed eight-sample batch and 77.4 seconds for the thinking configuration. Those waits are modelled from measured decode timings; neither is a recorded end-to-end service latency.

The counterexample is precise: for this pinned small model, this GSM8K subset, these seeds and this forced-closing protocol, simply granting more serial thinking tokens did not improve accuracy. It does not show that inference-time compute is ineffective in general.

The experiment is useful because it separates three ideas often compressed into “more reasoning”: letting an open model continue a serial thinking trace, drawing multiple stochastic answers, and stopping early when cheap samples agree. These mechanisms change the work, accuracy and latency in different ways. In this experiment, additional serial thinking did not help, while answer voting did; an adaptive voting rule subsequently reduced counted output tokens without changing observed accuracy on either saved sample set. That last result comes from offline reuse of eight already-generated samples, not from a deployed two-round generation experiment.

The chapter follows its plan by reporting accuracy, generated-token cost, and latency estimates for each method, including tokens and estimated seconds per correct answer. The accuracy and token counts come from actual local Qwen3 outputs; the wait figures come from a separate timing model. No hosted model was invoked, no provider charges were incurred, and no hidden provider reasoning was accessed. This distinction is essential when deciding which findings are measurements and which are projections.

The question

Inference-time, or test-time, compute means work added at inference without retraining the model: longer generated reasoning, multiple sampled completions, verifier-guided search, revision, or another allocation policy. Snell et al. compare verifier-guided search and iterative refinement and find that their relative value depends on problem difficulty. DeepSeek-R1 instead studies a reasoning capability developed through reinforcement learning; it does not establish that an arbitrary smaller model benefits monotonically from longer traces. Self-Consistency (Wang et al., ICLR 2023) motivates sampling diverse solutions and aggregating their answers. s1: Simple test-time scaling (EMNLP 2025) is a closer methodological comparison to this chapter’s forced-ending intervention, but uses a specifically fine-tuned much larger model and a different task distribution.

These papers are context for choosing experiments, not evidence for the numerical results below. The chapter does not reproduce a trained verifier, a search algorithm, or their reported performance improvements.

Four mechanisms can be examined locally without a learned verifier. Serial thinking generates a reasoning trace before a final answer. Sampling and voting generate several non-thinking completions and select the modal extracted answer. Budget forcing truncates an already-generated thinking trace and supplies a fixed continuation instruction; unlike an internally trained budget controller, this can alter the model’s input distribution. Adaptive allocation uses agreement among cheap samples to decide whether to spend more tokens. The experiments use actual model outputs for the first three and an offline decision rule for the fourth.

The engineering questions are causal and economic, not just numerical. For the same problems, which method changes the probability of a correct answer? How many generated tokens and how much estimated serving time does each require? Do additional samples improve accuracy beyond the first? Can disagreement act as an inexpensive escalation signal without looking at the gold answer? Does saving tokens also save wall-clock latency when the extra samples would otherwise run concurrently? Finally, what would a hosted-model control and accounting record actually let an application verify? The following sections answer these for a fixed experiment, with the limits of each comparison stated beside it.

Where this sits

Prefill and Decode distinguished first-token latency, token-to-token latency and throughput; reasoning adds output tokens even when none are shown as final answer text. Memory Bandwidth and Compute Bottlenecks and Continuous Batching explain how batching can amortise some decoding costs but also why larger batches and longer contexts eventually increase step time. Model Architecture Versus Runtime Architecture’s model capability records should include whether reasoning can be enabled, disabled or controlled, together with the tested template and tokenizer revision. Inference Protocols and Harmony described message and channel framing; that is not interchangeable with Qwen3’s local <think> convention or with a hosted provider’s private reasoning. Claude, Caching and API-Level Optimization introduced dated input/output pricing and a gated provider harness; it also identified the danger of treating character-based cost estimates as enforceable spending limits. This chapter reuses those gates for request admission in its unrun adapter, not as proof that a multi-request experiment has a hard budget cap.

Where the starter was wrong

The starter assigned success whenever a token budget exceeded the length of a task’s text. It therefore built rising accuracy into the simulator and charged the whole budget even if a task would have finished earlier. Its nine task/budget combinations produced 0.556 accuracy and 0.051 synthetic cost through arithmetic, not model behaviour. That code and its original tests remain as a historical baseline. The results below use sampled continuations of a pinned real model, saved per-problem outputs, an explicit answer extractor, and a separate latency model. The transition from the toy to the lab matters: the real curves can be flat, non-monotonic or worse at higher budgets.

The setup

Model and environment. The local model is Qwen/Qwen3-0.6B, revision c1899de289a04d12100db370d81485cdf75e47ca, under Apache-2.0, loaded in float16 with Transformers 5.17.0 and PyTorch 2.6.0+cu124. Experiments used one RTX 2060 12 GB on Windows 11, shared with other processes. The model card gives 28 layers, eight KV heads and a 32,768-token context. The harness enables thinking through Qwen3’s chat template and recognises the </think> token; its non-thinking condition sets enable_thinking=False. Sampling follows the card’s cited temperature/top-p/top-k configurations: 0.6/0.95/20 for thinking and 0.7/0.8/20 for non-thinking. This is not greedy decoding. The sampled outputs depend on checkpoint, tokenizer, seeds, generation implementation and batch composition; model identity alone is insufficient to reproduce them.

Tasks, splits and scoring. GSM8K supplies 1,319 test problems in the local dataset snapshot. The first 40 are designated development problems; twelve were used while debugging the harness. The primary evaluation sample consists of 200 distinct problems selected with seed 2026 from the other 1,279, and a separate 200-problem confirmation sample is selected after excluding both development and evaluation ids. The confirmation set was used to check the adaptive-voting rule proposed after evaluating the first set. Correctness is an automated numeric exact match: take the last complete \boxed{...} if present, otherwise the last number in the final 200 characters, normalise limited numeric notation, and compare with the reference. This extractor can misread answers that omit a box or include a later incidental number; its agreement with human grading was not audited. The samples are from one benchmark’s test split, not multiple independent tasks or model families.

Budget intervention. For each problem, the harness samples one thinking trace up to 1,536 generated tokens. A budget B keeps the initial B tokens of that trace; if the model has already emitted </think> by B, it continues naturally. Otherwise the harness appends its own fixed text—“I have run out of thinking budget, so I must answer now."—followed by </think> and requests an answer, capped at 256 newly generated tokens. Thus the treatment is prefix truncation plus a scripted instruction, not the natural output of a model trained to obey a token budget. Each problem’s five budgets share the same sampled thinking-prefix trajectory, which helps paired comparisons of the budget intervention but leaves stochastic answer continuations and batch context as possible sources of variation. There is no unforced completion beyond 1,536 thinking tokens against which to assess all cut cases.

def apply_budget(trace, budget, close_ids):
    if trace.finished and trace.thinking_tokens <= budget:
        return list(trace.tokens), False, trace.thinking_tokens      # it finished in time
    return list(trace.tokens[:budget]) + list(close_ids), True, min(budget, trace.thinking_tokens)

Non-thinking sampling. For every evaluation problem the harness creates eight sampled completions with thinking disabled, each capped at 400 tokens. A vote of k is retrospectively computed from the first k saved completions. Missing numeric answers are excluded from the ballot; ties choose the first answer among those tied, and an entirely empty ballot is wrong. The test therefore measures an answer-consistency heuristic, not a trained verifier. The generation wrapper seeds each length-sorted batch, rather than creating one independent, order-invariant generator per request: a different runtime batching policy may change the sampled answers. Reported gains are conditional on this fixed draw set.

Counting and uncertainty. A sample’s counted tokens are generated output tokens: thinking tokens plus its final answer, or all generated non-thinking answers used in a vote. The inserted forcing instruction, re-encoded input prefix, prefill/replay computation, and any provider-specific input charges are not included in the headline token totals. For a method with N tasks and C correct answers, “tokens per problem solved” means total counted generated tokens divided by C; it is not the average tokens of successful tasks. The accuracy intervals are 95% Wilson binomial intervals, while contrasts use paired bootstrap resampling of the same problems. The data include one set of stochastic draws, so the intervals measure variation over sampled problems conditional on those draws, not the variability from rerunning generation with different seeds.

How long it thinks

How long it thinks

On the 200 evaluation problems, 133 thinking traces emitted </think> within 1,536 generated tokens; 67 were right-censored at the cap. Among the 133 completed traces the median thinking length was 723 tokens and the 90th percentile 1,281. The cumulative counts completing by 128, 256, 512, 1,024 and 1,536 tokens were 0, 0, 25, 98 and 133. If all unfinished traces are assigned the 1,536-token cap, the mean observed/censored length is 1,058 tokens. This is not an estimate of the unconstrained mean thinking duration, since the 67 unfinished traces could have continued substantially longer. The experiment deliberately chose a lower cap and smaller batches after an earlier 2,048-token, 16-sequence run exhausted practical GPU capacity and produced no usable data.

The lab also measured a narrow repetition proxy: the share of eight-token sequences in each trace’s last 400 tokens that had appeared earlier in the trace. The observed means were 8% for traces that finished and 27% for those that reached the cap; four of the 67 unfinished traces exceeded 50%. That does not establish that the remaining traces were productive reasoning or “not stuck”: repetition can take other forms, and a long unique trace can still be wrong or stalled. No semantic annotation or faithfulness assessment of the thinking text was conducted. The safe inference is that overt repeated eight-grams were more common in unfinished traces.

Accuracy against thinking budget

Accuracy vs thinking budget

thinking budget correct (of 200) accuracy (95% Wilson interval) mean counted output tokens counted tokens per correct answer traces cut correct among cut correct among uncut
128 107 53.5% (46.6–60.3) 319 596 200 53.5% —
256 104 52.0% (45.1–58.8) 450 865 200 52.0% —
512 109 54.5% (47.6–61.3) 706 1,295 175 49.7% 88.0%
1,024 98 49.0% (42.2–55.9) 1,068 2,180 102 29.4% 69.4%
1,536 97 48.5% (41.7–55.4) 1,285 2,650 67 17.9% 63.9%

Each budget uses a prefix of the same capped thinking trace for a given problem. The cut/uncut columns condition on how long that trace happened to run and are not randomised subgroups.

From 128 to 512 tokens the accuracy estimates vary little relative to their uncertainty. The 1,024- and 1,536-token settings scored lower, but this does not establish a general decreasing law. On the same problems, 1,536 minus 1,024 tokens changed accuracy by −0.5 percentage points, with a paired bootstrap interval from −4.5 to +3.0 points. The cut/uncut columns cannot explain a causal benefit of thinking: a problem remains in the cut group precisely because its trace did not finish by that budget, a property correlated with difficulty and model behaviour. An 88% accuracy rate among the 25 traces that finished by 512 tokens is selection of a relatively easy subgroup, not a reward for the budget.

Counted output tokens per correct answer rise from about 596 at budget 128 to 2,650 at 1,536, because the extra continuation tokens do not correspond to more solved problems in these saved runs. This ratio is an output-token efficiency measure, not total inference FLOPs, monetary cost or latency. It omits extra prompt-processing work caused by constructing the forced answers from a regenerated context.

Parallel samples, thinking off

Parallel samples, thinking off

samples voted accuracy (95% Wilson interval) mean counted output tokens counted tokens per correct answer
1 55.5% (48.6–62.2) 263 474
2 55.5% (48.6–62.2) 531 957
3 59.5% (52.6–66.1) 796 1,338
4 61.0% (54.1–67.5) 1,059 1,736
8 62.5% (55.6–68.9) 2,114 3,383

These are votes computed from the first k of eight sampled completions per problem. More samples require more total generated tokens even where batching keeps the longest-sequence latency similar.

The individual non-thinking sample positions scored from 50.0% to 58.5% on this evaluation set. Using all eight improved the first-sample accuracy from 55.5% to 62.5%: a paired difference of +7 percentage points, with a 95% bootstrap interval of +2.5 to +11.5. The vote solved 19 problems missed by sample one and lost five it had solved. A vote of two did not improve the first sample here because disagreements are ties broken in favour of the earliest answer. These outcomes depend on the sampling seeds, answer extractor, tie-breaking convention and capped outputs; the bootstrap does not capture rerun-to-rerun sampling uncertainty.

Of the 1,600 non-thinking completions, 1,452 contained a recognised \boxed{...} and 161 reached the 400-token limit without an end token. A completion without a box may still contribute its last parsed number; a truncated completion may therefore receive a correct or incorrect numeric score even though it did not finish. The same extractor and cap apply to each sample position, but they can influence the size of the voting gain. A useful replication would report a strict “finished and boxed” accuracy beside this more permissive exact-number score.

The most decisive comparison on the evaluation problems is eight non-thinking samples versus a 1,024-token thinking budget: 125 correct against 98. Thinking minus voting was −13.5 percentage points (paired bootstrap interval −20 to −7); 37 problems were right only under the vote, and 10 only under thinking. However, those methods do not have equal token costs: 2,114 versus 1,068 counted generated tokens per task. The closer token comparison is thinking at 512 (706 tokens, 54.5%) versus a vote of three (796 tokens, 59.5%). Their paired difference was −5 points, interval −11 to +1.5, which does not exclude zero. It is therefore appropriate to say the three-sample vote looked better at similar output-token cost, not that its superiority was established. Neither comparison matches total compute, input-prefill cost or repeated-sampling variance.

Going from one non-thinking sample to eight added approximately 1,851 generated tokens per problem and increased accuracy by seven points. Across 200 problems that is about 370,200 additional counted tokens for 14 additional correct answers, or roughly 26,400 extra tokens per net additional correct answer. By contrast, moving the thinking budget from 512 to 1,024 added about 362 counted tokens per problem and reduced the observed number correct by 11. Marginal efficiency is meaningful only where the extra compute produces a reliable positive gain; negative gains should be reported as such, not converted into a “cost per improvement”.

Does thinking help where it finishes?

The overall averages hide a selection effect. The model finishes thinking early on a non-random subset of questions; these questions also tend to be easier for other methods. To examine that pattern, the saved non-thinking answers are scored on the same problem ids in the completed and unfinished groups. This descriptive comparison controls for which problems are included, but it does not make thinking duration a random treatment, and it does not tell us whether an early-finished trace caused the answer to be correct.

subgroup defined by thinking trace thinking answer at 1,536-token cap one non-thinking sample vote of eight, non-thinking mean counted thinking-plus-answer tokens
completed within 1,536 (133 problems) 63.9% 66.2% 72.9% 1,038
not completed within 1,536 (67) 17.9% 32.3% 41.8% 1,777
completed within 1,024 (98) 69.4% 73.3% 80.6% 869
not completed within 1,024 (102) 28.4% 37.0% 45.1% 1,686

The subgroups are selected after observing the thinking trace. These are descriptive, same-problem accuracy rates, not effect estimates for allowing the model to finish.

Among the 133 traces that completed within the cap, thinking scored 63.9%, compared with 66.2% from the first non-thinking sample and 72.9% from eight-sample voting on those same problems. Among the 67 censored traces, the forced answer scored 17.9%, versus 32.3% and 41.8%. These observations support a narrow engineering hypothesis: interrupting this model’s thinking and appending the lab’s fixed instruction may be a poor way to impose a budget. They do not isolate a causal harm from “thinking itself”: thinking-on and thinking-off use different chat templates and sampling settings, the subset is selected by trace duration, and the forced completion uses authored text that is not a native model stopping rule. No alternate forcing prompt or matched decoding ablation was run.

The eight non-thinking samples also allow a retrospective difficulty grouping: count how many of those already-generated samples matched the reference answer. This is an oracle-style analysis, because constructing the groups uses the gold labels and all eight answers. A deployed escalation rule does not know which samples are right; the next section instead uses answer agreement among three samples as a label-free proxy.

fraction of eight saved cheap samples correct problems thinking at 1,024 counted thinking-condition output tokens output tokens for one cheap sample
none 44 6.8% 1,153 322
some, but fewer than half 37 24.3% 1,219 296
at least half, but not all 58 53.4% 1,092 255
all eight 61 90.2% 893 212

These groups use correctness known only after evaluation. They measure association between sample success and thinking outcomes, not a deployable difficulty predictor.

At the 1,024-token condition, the thinking run used 893 counted output tokens on average for the 61 problems all eight cheap samples solved, and 1,153 for the 44 none solved. It correctly answered only 3 of those hardest 44. These averages show that the fixed budget did not successfully target additional compute where it yielded higher accuracy. They do not imply identical work on every problem, since some traces naturally finish early, and they do not test whether a learned difficulty predictor would work. The group boundaries here use reference correctness, which was unavailable to the actual generation policy.

Spending where it is needed

Snell et al. motivate allocating test-time compute according to prompt difficulty rather than spending the same amount everywhere. The local experiment supplies one inexpensive deployable signal: agreement among the first three numeric answers. Agreement does not prove correctness—different samples can share the same error—but it allows a decision without a verifier or ground truth. Two rules are evaluated against the same saved sample pool; their design histories are different, which matters for interpreting the confirmation run.

Adaptive thinking (specified before evaluation). Generate three non-thinking samples. If all three produce the same non-empty numeric answer, stop and return it; otherwise use the separately recorded 1,024-token forced-thinking result for that problem. On the primary evaluation set this routed 109 problems to thinking. It is a retrospective combination of outputs generated independently under fixed seeds, rather than a live request router with measured two-stage latency. Its purpose is to ask whether a cheap disagreement detector improves where the thinking tokens are spent.

Adaptive voting (conceived after examining the evaluation outcomes). Use the same first three samples. If they agree on a non-empty answer, stop; otherwise use the remaining five of the eight samples that had already been generated and vote across all eight. This supplies a reproducible offline, counterfactual token-accounting result, not a measured adaptive serving run: under actual two-round generation the later five samples might differ because the model’s random numbers are seeded per batch, and the batch composition changes. The rule was subsequently assessed on a separate 200-problem confirmation set using the same eight-samples-first method. Both sets therefore test policy selection on saved draws; neither tests the real cost and quality of generating the second round only when needed.

method on the 200-problem evaluation set accuracy mean counted output tokens counted tokens per correct answer escalated problems
one non-thinking sample 55.5% 263 474 —
always vote across eight 62.5% 2,114 3,383 —
always force thinking at 1,024 49.0% 1,068 2,180 —
three samples, then forced thinking if disagreement 55.0% 1,425 2,591 109
three samples, then vote eight if disagreement 62.5% 1,591 2,546 109

Adaptive rows are offline decisions over saved outputs. The second-round tokens were generated during full evaluation and were not withheld at runtime when the first three agreed.

The first three samples agreed on a non-empty answer for 91 of 200 evaluation problems; those answers were correct 89% of the time. The remaining 109 went to the escalation branch. Adaptive thinking scored 55.0% overall, six points above always thinking at 1,024 (paired interval +2.5 to +10), but it used more, not fewer, counted output tokens: 1,425 versus 1,068, because the first three samples were always generated. It did not beat the one-sample baseline. Offline adaptive voting achieved the same 125/200 as eight-sample voting while attributing 25% fewer generated tokens to its selected actions. Equality occurred in this particular saved sample set; agreement among three does not mathematically guarantee that five further votes cannot reverse the answer.

The adaptive-voting rule was also applied to a disjoint 200-problem confirmation sample that was not used to propose it. The new problems provide a genuine held-out check of this rule’s accuracy and offline token arithmetic. The samples still came from an always-generate-eight run, however, so this is not an independent test of runtime two-stage sampling, nor a rerun with varied sampling seeds.

method on held-out confirmation problems accuracy mean counted output tokens counted tokens per correct answer
one non-thinking sample 62.0% 249 402
vote of three 66.5% 745 1,121
always vote across eight 71.0% 1,986 2,797
offline adaptive voting 71.0% 1,410 1,985

Held-out refers to problem ids. The adaptive decision is recomputed from the first three of eight pre-generated responses; only the selected responses are charged in its offline accounting.

On the confirmation problems, offline adaptive voting again matched eight-sample voting problem for problem: 142/200 correct, versus 124 for one sample. It attributed 29% fewer generated tokens to the policy and escalated 97/200 problems. This replication supports the idea that three-way agreement was a good early-stop signal for these saved outputs. It does not prove that unanimity after three is generally safe or that the same result would appear if only escalated problems drew more samples in separate second-round batches. That is the next experimental gate: run the live conditional policy with request-scoped RNG, report actual generated tokens, accuracy, first-token and completion latency, and compare against always-eight on the same problems and multiple seeds.

What the user waits for

Generated tokens are not elapsed time. The lab measured decode-step timing with Transformers’ local generation loop on the shared RTX 2060 at batch sizes 1, 3 and 8 and several context lengths. The timing grid calls generate with greedy decoding (do_sample=False), whereas the task evaluations sample, so this is not an exact timing match. Per generated step the grid observed roughly 68–77 ms at batch one, 65–69 ms at batch three, and at batch eight approximately 66, 84 and 141 ms for contexts of 300, 1,000 and 1,700 tokens. Prefill of a short prompt took about 73 ms. The table below uses these measurements to model a request’s wait from prompt prefill and the longest sequence in a concurrently run batch, interpolating step cost by batch and context. It is not a trace of an eight-sample production server. It also does not include queueing, network transport, worker contention, and the full cost of forcing an answer by resubmitting the thinking prefix.

strategy modelled mean wait modelled p90 wait modelled seconds per correct answer observed evaluation accuracy
thinking budget 128 21.5 s 26.1 s 40.2 53.5%
thinking budget 512 49.3 s 53.9 s 90.5 54.5%
thinking budget 1,024 77.4 s 94.0 s 157.9 49.0%
thinking budget 1,536 95.7 s 137.5 s 197.4 48.5%
one non-thinking sample 17.7 s 26.4 s 31.9 55.5%
vote of three, concurrent 20.5 s 27.5 s 34.4 59.5%
vote of eight, concurrent 20.4 s 26.3 s 32.6 62.5%
adaptive thinking, two rounds 66.3 s 121.4 s 120.5 55.0%
adaptive voting, two rounds 32.7 s 54.5 s 52.4 62.5%

The elapsed seconds are derived from a local timing model and token lengths, not measured end-to-end waits for each strategy. “Seconds per correct answer” sums modelled waits over problems and divides by correctly answered problems; it is not marginal latency.

In this fitted timing model, eight concurrently sampled answers take approximately as long as one, despite using about eight times as many generated tokens. This is plausible for a decoder whose per-step time changes little with small batch size, but it is not a universal scheduling fact: at longer contexts the measured batch-eight step became much slower. The eight-vote row has the highest observed accuracy and a favourable modelled mean latency in this table; it is not best on token efficiency. Offline adaptive voting saves counted tokens but is assigned a longer wait because escalated problems pay for two sequential rounds. The relevant engineering choice is a Pareto trade-off between accuracy, real GPU-seconds, output tokens, queueing behaviour and user-facing latency—not a simple rule that parallel tokens are free.

The timing model was checked on ten budget-512, single-stream generations: measured waits ranged from 15.7 to 52.2 seconds for 448–768 tokens, while the model overpredicted by 12% on average and 95% in the worst individual case. A temporarily faster GPU operating point explains the large outlier in the saved report, but it was not experimentally isolated. Ten checks of one configuration do not validate the extrapolations to eight-way parallel batches or two-round adaptive requests. This machine was shared with a desktop workload, and the Transformers loop is not a tuned continuous-batching server. The mean/p90 seconds should be treated as illustrative planning estimates until the full strategies are run and timed end to end on controlled hardware.

The provider side, which was not run

Hosted reasoning models expose control fields that change by model generation. Anthropic’s extended-thinking documentation, reviewed for this revision on 26 October 2026, describes three cases: supported Claude 4.5-and-earlier models accept manual thinking: {"type": "enabled", "budget_tokens": N}; Claude 4.6 accepts that form but deprecates it in favour of adaptive thinking; Claude 4.7-and-later models reject manual thinking with HTTP 400 and use thinking: {"type": "adaptive"} with output_config.effort. The documentation names newer model-specific exceptions, so a numeric generation check is an approximation of the support table, not a permanent API capability registry. In manual mode the ordinary rule requires at least 1,024 thinking tokens and a budget below max_tokens; the documentation also records a special exception for interleaved thinking, which the adapter does not implement. A manual budget is a target, not the same thing as this chapter’s hard trace truncation. Thinking tokens contribute to billed output and may be reported under usage.output_tokens_details.thinking_tokens; visible thinking text may be summarised rather than a full trace. Changes of thinking mode or budget can affect caching as described in Claude, Caching and API-Level Optimization. These are provider documentation claims, not observations made by the lab. Anthropic extended thinking is the appropriate source to check before using any specific model.

examples/ch33/reasoning_adapters.py implements thinking_mode and thinking_config for the generation ranges the project recorded on 22 October 2026, returns suitable request fields for familiar model ids, and rejects combinations it considers unsupported. Those tests run with fake responses. AnthropicAdapter builds a problem request, parses usage into ThinkingUsage, and exposes evaluate for accuracy, tokens, elapsed seconds and dollars per correct answer. However, its model-support logic is a hard-coded generation rule, not a live capability lookup; its estimated preflight spend uses three characters per input token and a one-request ceiling; and it delegates to Claude, Caching and API-Level Optimization’s opt-in checks, which were not established as a cumulative hard expenditure cap. The adapter’s default correctness callback also returns False unless the caller supplies a grader, so simply invoking it does not yield an informative accuracy number. A production qualification must pin model id and docs revision, verify request schema, supply an audited grader, meter spending cumulatively, constrain total requests, and record transport/provider errors separately. None of that was exercised against Anthropic here.

No provider request was made. Accordingly, the chapter has no hosted-model accuracy curve, actual thinking-token bill, returned thinking summary, streaming trace or provider latency measurement. Anthropic’s documentation distinguishes internal thinking tokens from the visible summarised thinking blocks; the two should never be treated as the same count. By contrast, this Qwen3 experiment has the model’s locally generated, tokenizer-visible <think> span, subject to the harness’s interpretation of start/stop tokens. Neither source demonstrates that those tokens faithfully explain the output. The provider adapter is a testable interface proposal, not evidence that hosted Claude responds to more compute in the same way as Qwen3-0.6B.

What the numbers do not show

  • Model and task scope. Only Qwen3-0.6B in float16 on a 200-problem GSM8K evaluation sample was used for the main comparisons. Different checkpoints, prompts, tasks and trained reasoning models may scale differently. The 200-problem confirmation set was used only for the voting policies.
  • Censored thinking. Sixty-seven of 200 traces had not completed by the 1,536-token cap. The full-trace natural-answer accuracy of those 67 was not measured. Longer budgets (including 4,000 tokens) remain untested.
  • Forcing is an intervention. A single saved reasoning trace per problem, an authored closing instruction and a 256-token answer cap define the budget experiment. The inserted instruction is not a model-native token-budget API and its tokens are excluded from the reported generated-token cost.
  • Sampling is conditional on saved draws. The generation wrapper seeds length-sorted batches rather than guaranteeing request-indexed random streams. The adaptive voting and adaptive thinking rows are recombinations of separately generated saved outputs; neither adaptive policy was deployed as a conditional online generator.
  • Confidence intervals are conditional. Wilson intervals and paired bootstraps quantify variation over these problems for this saved stochastic sample. They do not estimate the additional variance from repeating inference with independent seeds or changing batch schedules.
  • Accuracy extraction is automated. The last boxed expression, or a fallback number, determines correctness. No blinded manual audit of answers missing boxes or truncated completions was performed, and no causal faithfulness assessment of reasoning text was made.
  • Output tokens are not full cost. Re-prompting with the capped trace, forced-closing text, prefilling the context, padding, host time and memory allocation are not reflected in the headline counted-token measure.
  • Latency is modelled. A local GPU timing grid underlies the waiting times, and only ten single-stream predictions were checked. These figures are not observed request latency in a live eight-way or adaptive service.
  • Hosted provider is NOT_RUN. API schemas, thinking-accounting rules and prices are sourced from documentation; the provider adapter is only exercised with a fake transport. Claude, Caching and API-Level Optimization’s preflight estimate is not an enforceable budget guarantee.
  • No trained verifier or search. External test-time scaling and self-consistency papers investigate different models, prompts and mechanisms. Their results were not reproduced.

Inference Lab increment

examples/ch33/reasoning_lab.py contributes a reproducible local reasoning experiment, not a serving engine or a model trained to reason under hard budgets. It provides the following parts, each of which should be matched to the saved raw outputs when re-running the chapter:

  • Dataset and score: load_gsm8k, split_problems, confirmation_set, last_boxed, normalise_number, extract_answer. Keep the dataset snapshot, split ids and extraction policy in the run manifest.
  • Generation: Qwen, Trace, run_traces, run_answers, run_vote, apply_budget. Persist the exact checkpoint/tokenizer revision, sampling parameters, cap, seeds, batch ordering and forced-ending text. The sample generator currently uses per-batch seeds. Manifest defect to repair: cmd_run records seeds.vote=13 in the confirmation metadata even though which="confirm" actually samples with seed 14; the raw-result evidence and the manifest must agree before claiming byte-reproducibility.
  • Comparison: wilson, paired_bootstrap, majority, vote_rows, adaptive_rows, adaptive_vote_rows, analyse, analyse_votes. These functions summarise saved outputs and offline decisions; they do not issue conditional second-round model calls.
  • Performance: LatencyModel, interp_step_time, repetition_share. Keep measured timing-grid observations separate from derived end-to-end waiting estimates.
  • Commands: inspect --help for the supported run and latency modes, and regenerate research/ch33/results.json only from retained raw evidence rather than overwriting the published result after changing the protocol.

examples/ch33/reasoning_adapters.py provides thinking_config, thinking_mode, ThinkingUsage, AnthropicAdapter, Result and evaluate. The supported-model table is encoded as broad generation ranges, the fake transport is the only exercised provider path, and a real evaluation requires a supplied correctness callback. Before permitting paid experiments, add accurate request-bound cost accounting, an experiment-wide spending reservation, explicit request and concurrency limits, model-specific schema tests, and failure accounting for partial responses.

To reconstruct the published accuracy and output-token tables, analyse the retained raw token outputs first. Generating a fresh run creates a new stochastic sample; it should receive a new run id rather than silently overwrite the results cited in the manuscript. The commands below indicate the recorded entry points, but the published experiments were expensive on the shared card and their duration is not a runtime guarantee.

python research/ch33/build_results.py research/ch33/latency.json
python examples/ch33/reasoning_lab.py run --which eval

The practical rule is: do not assume that increasing the reasoning budget increases correctness. On this pinned 0.6B model, serial forced thinking did not outperform cheaper sampled answers. Always-eight voting gave a statistically supported gain over one sample on the evaluation set, at considerably greater output-token cost. A three-sample agreement gate offered a promising offline way to save counted tokens, reproduced on disjoint problems; its real two-round generation, randomness and latency must still be qualified. Measure all of these on the target model and workload, in terms of task success, end-to-end waiting time, tokens, GPU resource use and error bars. A budget is an experimental variable, not a quality setting to maximise.

What you should now be able to do

  • Distinguish serial reasoning, forced trace truncation, parallel answer sampling, verifier-guided search, and adaptive test-time compute; identify which were actually tested.
  • Reproduce the 200-problem GSM8K splits and understand how fixed token extraction affects accuracy.
  • Explain why a single capped thinking trace shared across budgets supports some paired comparisons but not claims about unconstrained long reasoning.
  • Compute Wilson intervals, paired differences and generated tokens per correct answer without confusing output-token cost with full serving compute.
  • Implement deterministic vote and tie semantics and distinguish a saved-output policy evaluation from live conditional sampling.
  • Explain why a small batch can reduce user wait at the cost of more total generated tokens, and identify where the latency model was and was not validated.
  • Build a provider request from versioned model capabilities and understand why API-reported thinking tokens and summarised thinking text differ.
  • State an evidence gate for the next iteration: more seeds, audited answer extraction, real two-stage generation, measured end-to-end timings and enforceable provider budget control.

Key terms

  • Inference-time compute: extra generation, search, checking or revision performed at inference using fixed model parameters.
  • Thinking trace: locally generated tokens in a model’s reasoning span; not necessarily a faithful explanation of its answer.
  • Thinking budget: a chosen limit or target for reasoning expenditure; here enforced by truncating a trace and supplying a closing instruction, unlike a provider-native soft budget.
  • Budget forcing: an intervention that ends or prolongs a thinking trace by steering generated text; results depend on the model and intervention.
  • Majority vote / self-consistency: selection of the most frequent extracted answer from multiple stochastic samples; this lab breaks ties in favour of the first encountered answer.
  • Tokens per correct answer: all counted generated tokens divided by the number of correct problems, not only the tokens for correct outputs.
  • Marginal cost: additional compute or tokens divided by the additional number of correct problems, meaningful when a reproducible positive gain is present.
  • Paired comparison: the difference between methods on the same question ids; pairing does not eliminate stochastic seed or batching effects.
  • Adaptive allocation: a policy deciding which requests receive additional inference work, based on signals available when the decision is made.
  • Offline policy evaluation: apply a proposed decision rule to fully generated saved data and charge only outputs it would have selected; this is not an online deployment test.
  • Effort level: a hosted-model configuration giving soft guidance to reasoning depth rather than a guaranteed thinking-token count.

Reader exercises

  1. Reproduce the evaluation and confirmation splits from the recorded dataset snapshot; store hashes of the problem ids and raw outputs.
  2. Repeat the non-thinking and thinking conditions across at least five independent request-scoped seeds and report between-seed as well as between-problem uncertainty.
  3. Raise the thinking cap to 4,000 tokens on a dedicated GPU and report how many formerly censored traces finish and whether their answers improve.
  4. Compare at least two forced-closing instructions selected on development data, then lock the protocol before testing it on a new held-out set.
  5. Run true two-stage adaptive voting: produce three samples, decide, and generate the remaining five only on disagreement, using request-indexed random streams; compare the resulting accuracy and token accounting with the saved-output counterfactual.
  6. Manually audit a blinded sample of boxed, unboxed and truncated outputs and quantify extraction false positives and false negatives.
  7. Add a verifier to rerank candidate solutions, and compare verifier cost with majority voting at equal generated-token and measured latency budgets.
  8. Serve the same Qwen3 model using KV reuse between thinking and answer; record prefill, decode, peak memory, queue wait and end-to-end completion time for each condition.
  9. Extend the provider adapter with a pinned model capability table, an audited grader and a real experiment-wide hard spending gate; require explicit authorisation before sending requests.
  10. Add an inference-run manifest asserting dataset ids, model/tokenizer revision, thinking template, forced text, seeds, batch order, scoring, tool versions, raw evidence hashes and cost-accounting scope.

References

Evidence in this chapter

Claim Evidence classification
Qwen3-0.6B reached 53.5%, 52.0%, 54.5%, 49.0% and 48.5% at increasing forced-thinking budgets over the same 200 GSM8K evaluation questions MEASURED: one capped thinking trace per question, with scripted closures and sampled answer continuations; not unconstrained reasoning
The first non-thinking sample scored 55.5%, the first three voted to 59.5%, and eight voted to 62.5% MEASURED: saved local generations; the sample count and deterministic tie rule are fixed
Eight-vote minus 1,024-token thinking was +13.5 points, with a paired interval excluding zero DERIVED FROM MEASURED OUTPUTS: paired over problem ids, conditional on sampled draws
Completed thinking scored 63.9% while one non-thinking sample scored 66.2% on those same questions DESCRIPTIVE: subgroup conditioned on thinking completion, not a causal effect
Three-way agreement allowed offline adaptive voting to match eight votes while attributing 25% fewer tokens in evaluation and 29% fewer on confirmation problems OFFLINE POLICY EVALUATION: selected from eight already-generated outputs; online second-round behaviour NOT_RUN
The eight-vote and 1,024-token thinking conditions would take mean waits of 20.4 s and 77.4 s under the local timing model MODELLED: based on measured decode timings; not end-to-end server measurements
A forced thinking prefix plus closing text yields the same outputs as a native per-request thinking-budget API NOT ESTABLISHED: the local intervention differs from provider controls
Hosted models accept different manual/adaptive controls and report thinking tokens in billed output DOCUMENTED: versioned provider documentation; fake-transport adapter tests only
Provider thinking accuracy, live spend, partial-stream behaviour, hardware-general latency, trained verifiers and reasoning faithfulness NOT_RUN / NOT_EVALUATED

Implementation status

  • Manuscript: content/books/inference/33-chapter.md.
  • Code: examples/ch33/reasoning.py remains the historical simulator; examples/ch33/reasoning_lab.py implements local Qwen generation, answer extraction, budget forcing, voting and saved-output analysis; examples/ch33/reasoning_adapters.py implements unrun provider request/usage handling and a common evaluation interface.
  • Tests: tests/test_ch33_reasoning.py and tests/test_ch33_reasoning_lab.py. The latter records 38 test cases covering extraction, intervals, voting, scripted adaptive rules, dataset splits, batching and stop logic on a tiny CPU Qwen where the tokenizer is available, plus fake-transport provider controls. This is the repository’s recorded coverage, not a claim that those tests were re-executed during editorial review.
  • Evidence: research/ch33/results.json, its builder and research/ch33/raw/ hold the saved generated-token runs. The raw inputs, checkpoint revision and environment should remain immutable when a new experiment is added. A code audit found that confirmation generation uses vote seed 14 but its common metadata writer records 13; fix this manifest mismatch without rewriting the previously recorded results.
  • Verification: python -m unittest discover -s tests -v is the repository’s documented suite command; an independent verification should record execution date, revision, skips and output instead of inferring a pass from this manuscript.

The chapter is published as a draft (draft = false) and demonstrates one strong, bounded negative result: more forced serial thinking did not improve this small model’s GSM8K accuracy in the tested range. Its positive voting results come from real saved model outputs; its adaptive savings are offline-policy results, and its elapsed-time values are projections. Before generalising to a serving recommendation, run an actual two-stage adaptive service with stable request-level randomness, independently check answer extraction, and measure complete cost and latency. Tool Calling and Agentic Inference can then price the observed serving workload rather than treating counted output tokens as the whole bill.