Inference From First Principles on Programmer.ie: Modern AI programming

What Is Inference?

Tue, 20 Oct 2026 00:00:00 +0100

I asked a 124-million-parameter model to continue a sentence by sixteen tokens. On a CPU it took about 2.7 seconds. The answer was:

An inference engine takes a trained model and a request, and it must turn them into a model that can be used to generate a model that can be used to generate

Three things are true of that result at once. It was slow for something that produced a few dozen characters. It was repetitive to the point of being useless. And it was entirely correct, in the sense that the program did exactly what greedy decoding of that model is defined to do. Run it again and you get the same sixteen tokens.

From Training to Execution

Tue, 20 Oct 2026 00:00:00 +0100

The GPT-2 file on my disk is 548,105,171 bytes. The model it contains has 124,439,808 parameters, and each is a 32-bit float, so the parameters occupy 497,759,232 bytes. About 50 megabytes of the file are not parameters at all.

That gap is a small thing, and it is the whole subject of this chapter in miniature. We say “the model” as though it were one object, a block of learned numbers. What actually arrives from training is a bundle of things: tensors that were learned, tensors that were not, a description of how to wire them together, a tokeniser, default generation settings, and the code that interprets all of them. An inference system has to take that bundle and turn it into something that executes. Between the two lie decisions that nobody made explicitly, and several of them change what the model does.

Tokens, Embeddings and Probability

Tue, 20 Oct 2026 00:00:00 +0100

Chapter 1 ended with a sentence that looped. Forty greedy steps from the same prompt produce this:

An inference engine takes a trained model and a request, and it must turn them into a model that can be used to generate a model that can be used to generate a model that can be used to generate a model that can be used to generate a model that can be used to generate

The Transformer as a Computation

Tue, 20 Oct 2026 00:00:00 +0100

I wrote GPT-2’s forward pass out by hand. It is about forty lines of tensor operations: an embedding lookup, twelve repetitions of a block built from layer norm, attention and a two-layer feed-forward network, a final layer norm, and a matrix product. Fed the weights of the real model, it returns logits that agree with the library’s to within \(1.2 \times 10^{-4}\) on logits as large as 323 in magnitude, and picks the same most probable token at every position of four different prompts.

Autoregressive Generation

Tue, 20 Oct 2026 00:00:00 +0100

Generating 64 tokens from GPT-2 on my CPU took 11.5 seconds. I then took the 64 tokens it had produced, appended them to the prompt, and ran the model once over the whole 81-token sequence. That took 0.42 seconds, and at every one of the 64 positions the model’s most probable next token was exactly the token the slow loop had chosen.

Producing the answer was 27 times slower than checking it. That asymmetry is the most important structural fact about generation, and the rest of Part III is largely a sequence of ways to exploit it. We should be careful about the number, and I will qualify it below, because the slow loop is a naive one and a competent loop would narrow the gap. But the direction is not in doubt, and it rests on a distinction that is easy to blur: autoregressive generation is serial in which tokens are chosen, and parallel in how tokens are scored.

Building Our First Inference Engine

Tue, 20 Oct 2026 00:00:00 +0100

The sampler in this chapter’s original engine had a bug that no test caught and no output revealed. Step \(k\) of a request with seed \(s\) used the random seed \(s + k\). That looks harmless, and it means that request 2’s step 0 draws the same random number as request 1’s step 1, request 3’s step 0 the same as request 2’s step 1, and so on. For 100 requests of 32 steps each, 3,200 draws needed 3,200 independent random streams, and the scheme provided 131. When I fed two requests the same scores at every step, the output of seed 2 was an exact one-step-shifted copy of seed 1 in all 31 compared positions.

Matrix Multiplication and GPU Execution

Tue, 20 Oct 2026 00:00:00 +0100

The engine from Chapter 6 generates 32 tokens from GPT-2 in 6.4 seconds on one CPU thread. Move the same weights to the GPU in this machine and it takes 0.45 seconds. The output is the same 32 tokens. That is a 14-fold improvement, and it should feel too small.

The same GPU multiplies two large matrices at about 7,000 billion floating-point operations per second. One CPU thread manages about 53 billion. That is a ratio of roughly 130. So where did the other 90 per cent of the advantage go? Part of the answer is that a language model is not one large matrix multiplication. It is hundreds of small ones, and small ones behave differently. A single-token step through the output layer of GPT-2 uses about 2 per cent of the arithmetic this GPU can deliver. The chip spends the rest of that time waiting for weights to arrive from memory. Another part of the answer is that, below a certain size, a GPU call costs about 35 microseconds whatever it computes, and a CPU can multiply two small matrices in a tenth of that.

Attention and Its Computational Cost

Tue, 20 Oct 2026 00:00:00 +0100

For a 16,384-token sequence with GPT-2’s twelve heads in half precision, one attention score tensor is 6.0 GiB. A straightforward implementation of attention needs two of them at once. On this machine it asked for 12.3 GiB, on a card with 12, and did not fail. I will explain below why it did not, because the explanation is itself instructive. PyTorch’s own attention function, given the same inputs, used 24 MiB, which is the size of its output. The two agreed on the result to within the rounding error of 16-bit floating point, and the second was eleven times faster at a length of 8,192.

Prefill and Decode

Tue, 20 Oct 2026 00:00:00 +0100

On this machine, GPT-2 takes 12 milliseconds to read a 16-token prompt and produce its first token, and 11 milliseconds for each token after that. Give it a prompt of 1,000 tokens and the first token takes 44 milliseconds, but the tokens after it still take 11. A 64-token reply to the short prompt spends 98 per cent of its time in the second phase. A 4-token reply to the long prompt spends 56 per cent in the first.

Key-Value Caching

Tue, 20 Oct 2026 00:00:00 +0100

I reused a key-value cache built for one prompt to continue a different one. Nothing raised an error. The next-token scores were off by 11.6 on the median across twenty trials, and by 28.7 in the worst, and the model picked the right next token in only 40 per cent of them. In a second experiment I built the cache with weights that differed from the real ones by 2 per cent in a single layer, as an adapter swap might. The scores were off by 0.04, and the right token came out in 20 trials of 20.

Memory Bandwidth and Compute Bottlenecks

Tue, 20 Oct 2026 00:00:00 +0100

I wrote down a prediction for how long 58 different GPU operations should take, using nothing but their size and three numbers measured on this machine. The median measurement came out 10 to 14 per cent above the prediction. At least nineteen in twenty were within a factor of two. And not one memory-bound operation ran faster than its bound, which is what a bound is for.

Then I used the same model to predict the time of a whole decode step as the batch size grew, and it was wrong by a factor of three. It said the step would stay at its floor until a batch of around 256. The measured step started to climb at 64, and at 256 it was 30 milliseconds against a prediction of 10.

Quantization and Numerical Precision

Tue, 20 Oct 2026 00:00:00 +0100

I quantised the same matrix, the 38.6-million-entry embedding table of GPT-2, to four bits in two different ways. Applied to the input side, where tokens are looked up, it cost 0.7 per cent in perplexity. Applied to the output side, where the same numbers produce the scores for the next token, perplexity went from 61 to 8,123. The same matrix, the same error of about 13 per cent in its weights, and a difference of four orders of magnitude in the consequence.

FlashAttention and Optimized Kernels

Tue, 20 Oct 2026 00:00:00 +0100

I wrote exact block-wise attention in plain PyTorch, the algorithm that FlashAttention is built on. It gave the same answer as ordinary attention to within rounding. At 8,192 tokens it needed 16 MiB of extra memory where ordinary attention needed 3,148. And it was sixteen to eighteen times slower.

The version of the same algorithm that PyTorch ships as a fused kernel, running on the same inputs, needed 12 MiB and, depending on length and run, was between 5.5 and 13.5 times faster than ordinary attention.

PagedAttention

Tue, 20 Oct 2026 00:00:00 +0100

A server with room for 32,768 tokens of KV cache can hold exactly 32 sequences if it sets aside 1,024 tokens for each. In my simulation of a plausible chat workload, those 32 sequences used 33 per cent of the memory they had been given. The other two thirds sat reserved for tokens that, on average, would never be generated.

Give the same budget to an allocator that hands out 16-token blocks as sequences grow, and the server holds between 92 and 96 sequences at a time, using 98 per cent of the memory, and finishes the same 3,000 requests in 2.6 to 2.7 times fewer steps.

Continuous Batching

Tue, 20 Oct 2026 00:00:00 +0100

Under a light load, three tenths of the capacity of the server, requests to a statically batched GPT-2 waited a median of one to one and a half seconds for their first token. The same requests to the same model on the same GPU with continuous batching waited 55 to 60 milliseconds. At a load near capacity the median wait for a first token was five to six seconds for the static server with a batch of 8, and 1.4 to 2.4 seconds with a batch of 32. The continuous server’s was still 60 to 70 milliseconds.

Prefix Caching

Tue, 20 Oct 2026 00:00:00 +0100

With a 512-token prefix already in the cache, a 32-token follow-up got its first token in 22 milliseconds. Without it, the same request took 37. A second tenant’s request, probing the same prefix, could tell the difference every single time: in 40 trials, the probe that hit a cached prefix was faster than every one that did not.

The cache is a speed-up, and the speed-up is a measurement anyone can make. If two tenants share a prefix cache, either can find out, by timing, whether the other has recently sent a particular prompt. Giving each tenant its own scope removes that signal. In the same experiment the scoped probe was indistinguishable from a cold one (the chance that a cold probe was slower than a scoped one was 55 per cent, close to a coin toss), and it costs a share of the hit rate that depends heavily on how big the cache is.

Speculative Decoding

Wed, 21 Oct 2026 00:00:00 +0100

The same 3-billion-parameter model, on the same GPU, produced tokens between 0.77 and 1.51 times as fast as it normally does, depending only on what it was writing and how many tokens a small helper model guessed ahead. On repetitive or structured text with six tokens guessed per step, it was 1.51 times faster. On paragraphs of prose with the same setting, it was slower than not guessing at all.

Compilation and Execution Graphs

Wed, 21 Oct 2026 00:00:00 +0100

One decode step of GPT-2 took 16.5 milliseconds when PyTorch ran it operation by operation, and 3.4 milliseconds when the same GPU work was recorded once and replayed. The tokens were identical, all 129 of them. Four fifths of the eager step was not the GPU computing anything; it was the host deciding what to launch next.

The previous draft of this chapter reported a different result: that compilation could not be measured at all, because TorchInductor “could not find a C++ compiler” on this machine. That was wrong. Visual Studio 2022 is installed; its compiler was simply not on the PATH of the shell that ran the test. The error was real, the conclusion drawn from it was not, and the chapter had been recording an environment accident as a fact about the machine. With the compiler’s environment loaded, Inductor works on the CPU, though not in the way I first expected, and on the GPU it does not work because a different dependency is missing. Section by section, this chapter separates three things that the word “compile” covers, measures what each does to a decode step, and counts what each costs before it pays.

Mixture-of-Experts Inference

Wed, 21 Oct 2026 00:00:00 +0100

Mixtral 8x7B computes each token with 13 billion parameters and stores 47 billion. That sounds like a model that is three and a half times cheaper to run than its size suggests, and for one token at a time it nearly is. On a layer built to Mixtral’s shape, one token chose two of eight experts and the layer took 3.95 milliseconds, reading only the experts it needed. Sixteen tokens chose between them all eight, and the same layer took 9.7 milliseconds, while a dense layer with the same active width took 1.75. The saving that sparsity promises is a saving per token. A batch pays for the union of what its tokens chose.

Expert Routing and Load Balancing

Wed, 21 Oct 2026 00:00:00 +0100

Three small mixture-of-experts language models were trained identically except for how they were encouraged to balance their experts. All three reached the same quality on held-out text, to within the difference between seeds. Then each was given the same fixed capacity per expert at inference, as a serving system with static shapes would, for batches of 64 tokens. The model with no balancing term lost 0.28 bits per character at a capacity factor of 2.0; the two balanced models lost 0.05. At a factor of 1.25 the loss was 0.68 bits against 0.33. The weights were not different in kind, nor was the quality without a limit. What differed was how evenly the router spread its tokens, and that is invisible until an execution engine with a finite budget meets it.

Multi-Head and Multi-Latent Attention

Thu, 22 Oct 2026 00:00:00 +0100

Llama-3.2-3B keeps 112 kibibytes of keys and values for every token of every sequence. Had it been built with a separate key/value head for each of its 24 query heads, instead of 8 shared ones, that would be 336. On the 12 GB card of this book, with the weights loaded, the first design leaves room for 11 sequences of 4,096 tokens and the second for 3.

That saving is a decision made when the model was designed, and a serving system can lose it without noticing. The way the book’s own starter code wrote grouped-query attention (repeat each shared key/value head for its queries, then run ordinary attention) reads the same bytes as the multi-head version. In the measurement below, going from 32 key/value heads to 1 changed its decode time by almost nothing (29 to 31 milliseconds). Written so that each cached byte is read once, the same step at Llama-3.2’s 8 key/value heads took 1.85 milliseconds instead of 30.

Sliding-Window Attention

Thu, 22 Oct 2026 00:00:00 +0100

Put a 2,048-token window on Llama-3.2-1B and score it on the second half of a 4,096-token passage. With full attention its score is 3.00 bits per token. With the window it is 7.07, more than twice as bad. Now let the window also keep the very first token of the passage, and the score is 3.35. One token, kept out of 2,048, takes back 91 per cent of the damage.

Long-Context Inference

Thu, 22 Oct 2026 00:00:00 +0100

Llama-3.2-1B advertises a context of 131,072 tokens. On the 12 GiB card of this book, asked to prefill 4,096 of them with the library’s default attention, it used 4.9 GiB of extra memory, nearly four times what 2,048 needed. A prefill of 8,192, tried once outside the final measurements, wanted nearly 19 GiB and took two and a half minutes because the driver quietly spilled the excess into system memory. The model accepted the tokens in every case. Nothing in “accepts 131,072 tokens” said that.

Model Architecture Versus Runtime Architecture

Thu, 22 Oct 2026 00:00:00 +0100

Seven things were changed about how Llama-3.2-1B was run, one at a time, on twelve prompts of 48 greedy tokens each: a chunked prefill (twice), a different attention kernel (twice), a static cache, a padded batch, and float32 arithmetic. In 79 of the 84 runs the output was identical to the baseline’s, token for token. The other five differed at a token where the model had been within 0.02 logits of a tie, three of them an exact tie, and the largest change to the logits at the end of any prompt was 0.027.

From Inference Loop to API

Tue, 20 Oct 2026 00:00:00 +0100

Twenty-nine awkward requests were sent to a first-draft inference service, the kind that takes the framework’s defaults: a body that is not JSON, a prompt that is empty, an option spelt wrongly, a temperature of NaN, a prompt of 4,000 emoji. It answered 200 to eight of them. Four of those answers were not what the caller asked for: "5", 5.0 and true were all accepted as a token limit, and a request for max_tokens: 2 (a misspelling of max_new_tokens) returned sixteen tokens, the default of the field that did exist. It answered 500 to four others, and none of its twenty-one errors came in a shape the service had chosen or documented: most were the framework’s detail lists, the rest one-line strings.

Request Scheduling and Admission Control

Tue, 20 Oct 2026 00:00:00 +0100

The same service was sent the same 279 requests in 40 seconds, at about twice the rate its engine could serve. A 16-token request took the engine about 0.3 seconds, and the arrivals brought about two seconds of work for every second of engine time. A client was happy with an answer within two seconds.

With an unbounded queue, which is what the Chapter 25 service had, 23 of the 279 requests were answered in time. The median request waited 26 seconds and the 99th percentile 57. Every request was eventually answered correctly; 256 of them were late. With a queue limited to eight, 48 were in time. With a queue that also dropped what could no longer finish by its deadline, 185 were. With an admission rule that refused, at arrival, what its own estimate said could not finish in time, 224 were, and no request that was served was late.

Concurrent Requests and Streaming

Tue, 20 Oct 2026 00:00:00 +0100

Thirty-two clients each asked for 64 tokens. Served one after another by the engine of Chapters 25 and 26, they would have taken 22 seconds in total, and the last would have waited 21 seconds for its first word. On the batched engine of this chapter they were all answered in about 1.5 seconds, 14 to 15 times faster, at 1,320 to 1,376 tokens a second against 90 to 94 for a single stream. Every one of the 189 concurrent streams I compared with its solo run had identical token ids.

Distributed Inference

Tue, 20 Oct 2026 00:00:00 +0100

GPT-2 was split across six processes, each holding a sixth of every layer’s attention heads and feed-forward width. Every one of the 15 runs of the split code, over 1, 2, 3, 4 and 6 processes, produced the library model’s 25 tokens exactly, and the final logits of the prompt differed from the library’s by at most 9.2e-5. The split model was also slower. One process took 23 milliseconds to produce a token. Six took 54 to 59.

Prefill/Decode Disaggregation

Tue, 20 Oct 2026 00:00:00 +0100

Chapter 28 scaled across devices. This chapter splits the two phases of generation across different workers.

Why disaggregate

Prefill and decode have different bottlenecks. Prefill is compute-bound; decode is memory-bound. Running both on the same hardware forces a compromise. Disaggregation assigns prefill to compute-optimized workers and decode to memory-optimized workers.

Transfer cost

Disaggregation requires moving KV state from prefill workers to decode workers. If transfer cost is high, it can erase the benefit. The simulator includes a transfer-cost counterexample.

Reliability, Cancellation and Recovery

Tue, 20 Oct 2026 00:00:00 +0100

Chapter 29 split phases. This chapter handles failures.

Failure modes

Inference services face OOM, worker loss, malformed inputs, and partial streams. Each requires explicit handling.

Cancellation

Cancellation stops work and releases resources. It must propagate to all stages. gRPC cancellation shows that signalling alone is insufficient; applications must propagate it and release their own resources.

Timeouts

Timeouts prevent unbounded work. They should be enforced at request and operation levels.

Retries

Bounded retries can recover from transient failures. They must not mask persistent errors or cause duplicate side effects. Retryable transport failure differs from unsafe replay of a non-idempotent request.

Inference Protocols and Harmony

Tue, 20 Oct 2026 00:00:00 +0100

Chapter 30 handled reliability. Part 6 now examines protocols and application-level optimization, starting with message envelopes.

Message envelopes

Inference protocols wrap raw text in structured messages with roles and channels. This allows system, user, assistant, and tool messages to be distinguished.

Roles and channels

Roles identify who produced a message. Channels distinguish analysis from final output. This separation supports reasoning and safety.

Tool handoffs

Tool calls require structured requests and responses. The protocol must preserve boundaries so untrusted content is not interpreted as instructions.

Claude, Caching and API-Level Optimization

Tue, 20 Oct 2026 00:00:00 +0100

Chapter 31 examined protocols. This chapter examines API-level optimization, using public documentation only.

Public API versus internals

Provider documentation describes request/response behavior, not serving internals. We must not infer proprietary implementations from API behavior.

Prompt caching

Prompt caching allows reuse of previously processed prefixes. Eligibility depends on documented fields such as minimum token counts and cache control markers. Savings are documented discounts, not measured system behavior.

Research context

Anthropic’s documentation defines prompt caching as an observable contract: cache breakpoints, TTLs, ordering, and usage accounting are visible. The Messages API remains stateless: the client still resends context, while provider-side caching may reduce cost. Pricing is published but volatile and contract-sensitive.

Reasoning Models and Inference-Time Compute

Tue, 20 Oct 2026 00:00:00 +0100

Chapter 32 examined API-level optimization. This chapter examines reasoning models that spend additional inference-time compute.

Reasoning tokens

Some models generate intermediate reasoning tokens before final answers. These tokens consume compute and cost but are not always visible.

Budget controls

Budgets limit reasoning length. Higher budgets can improve accuracy but increase latency and cost. The trade-off is task-dependent. More reasoning tokens do not always improve accuracy.

Research context

Inference-time compute includes serial reasoning, parallel samples, verifier search, and adaptive allocation. The cited sources study optimal test-time compute and DeepSeek-R1’s reasoning training. Claude’s extended thinking controls are model-specific and versioned; fixed budget_tokens is not universally supported.

Tool Calling and Agentic Inference

Tue, 20 Oct 2026 00:00:00 +0100

Chapter 33 examined reasoning budgets. This chapter examines tool use.

Tool calling

Tool calling lets a model request external computation. The runtime must parse the request, execute safely, and return results. The model proposes calls; the application is responsible for authorisation, execution, validation, and retries.

Interruption and re-entry

Generation pauses for tool execution, then resumes with the tool result. This requires state management and trace IDs.

Sandboxing

Tools must be sandboxed. Untrusted model output must not execute arbitrary code. Tool arguments and results are untrusted data requiring schemas and policy checks.

Structured Outputs and Constrained Decoding

Tue, 20 Oct 2026 00:00:00 +0100

Chapter 34 handled tool calls. This chapter ensures outputs conform to schemas.

Schemas

Structured outputs require a schema. The runtime must validate generated JSON against it.

Token-level constraints

Constrained decoding restricts tokens to those that keep the output valid. This can guarantee syntactic validity but not semantic correctness.

Validation and repair

Validation checks output after generation. Repair attempts to fix invalid output. Validation is safer; repair can introduce errors.

Research context

The selected sources distinguish grammar compilation, token masking, parser state, and schema validation. XGrammar and guided-generation work show that syntactic conformance can be efficient, but JSONSchemaBench shows that schema-valid JSON can still be semantically wrong or unsafe.

Latency, Throughput and Time to First Token

Tue, 20 Oct 2026 00:00:00 +0100

Chapter 35 structured outputs. Part 7 now examines economics and evaluation, starting with metrics.

Definitions

TTFT: time to first token.
TPOT: time per output token.
E2E latency: total request time.
Throughput: tokens per second.
Percentiles: tail behavior.

Why percentiles

Averages hide tail latency. Percentiles reveal user experience. TTFT p99 matters for responsiveness.

Denominators

Do not mix prompt and output tokens in throughput without stating. Prompt and output have different costs.

Cost per Million Tokens

Tue, 20 Oct 2026 00:00:00 +0100

Chapter 36 defined metrics. This chapter converts them to cost.

Cost components

Cost includes hardware amortization, power, cooling, networking, storage, and operational overhead. API billing adds provider margin and abstraction.

Utilization

Utilization is critical. Low utilization raises cost per token. The estimator shows sensitivity to utilization.

Forecasts versus measurements

Estimates are forecasts based on assumptions. Measured spend requires actual billing and usage data.

Research context

Separate API list price from measured self-hosted cost. Build cost from token mix, cache hits, batch efficiency, utilisation, hardware time, energy, and rejected or aborted work. Do not compare providers without model, quality, region, contract, and input/output assumptions. Pope et al. model inference cost and latency for large models, but their measurements are for specific models and TPU v4 configurations.

GPU Utilization and Capacity Planning

Tue, 20 Oct 2026 00:00:00 +0100

Chapter 37 estimated cost. This chapter plans capacity.

Occupancy

GPU occupancy measures how many warps are active. Low occupancy can indicate underutilization or memory bottlenecks.

Memory limits

Memory limits cap batch size and context length. Exceeding memory causes OOM.

Queue saturation

Queue saturation occurs when arrival rate exceeds service rate. Latency grows without bound.

Capacity headroom

Capacity planning requires headroom for bursts. Extrapolating beyond measured load is unsafe. The model rejects loads beyond observed.

Correctness, Determinism and Reproducibility

Tue, 20 Oct 2026 00:00:00 +0100

Chapter 38 planned capacity. This chapter examines correctness and reproducibility.

Floating-point nondeterminism

Floating-point operations are not associative. Parallel reductions can produce different results. GPU kernels may vary by version and hardware.

Batching effects

Batch size can affect numerical results due to reduction order. Batch invariance is not automatic.

Seed limitations

Seeds help reproducibility but do not guarantee identical results across hardware or library versions.

Research context

Separate seeded repeatability, deterministic algorithms, batch invariance, numerical tolerance, and semantic equivalence. Record model/tokenizer revisions, runtime and driver versions, dtype, kernels, hardware, request ordering, and sampler state. Cross-hardware bitwise identity is a stronger and often unavailable guarantee. PyTorch documents that deterministic algorithms can reduce performance and still not promise cross-version identity.

Designing Meaningful Inference Benchmarks

Tue, 20 Oct 2026 00:00:00 +0100

Chapter 39 addressed reproducibility. This chapter designs benchmarks.

Realistic workloads

Benchmarks must reflect real usage: prompt lengths, output lengths, concurrency, and task types.

Baselines

A benchmark needs a baseline. Compare against a simple, well-understood implementation.

Statistical uncertainty

Repeated trials and confidence intervals quantify uncertainty. Single runs are insufficient.

Research context

Freeze the system under test, model, revision, quality target, workload, arrival process, cache state, and measurement points. Report TTFT, TPOT, ITL, end-to-end percentiles, throughput, goodput, errors, and power where relevant. MLPerf separates scenarios, quality targets, latency constraints, throughput, system metadata, and power. vLLM documents cache-related benchmark hazards. Separate repository-reported reference graphs from measurements reproduced by this book.

Designing the Inference Runtime

Tue, 20 Oct 2026 00:00:00 +0100

Chapter 40 designed benchmarks. Part 8 now builds the Inference Lab capstone, starting with design.

Architecture

The runtime integrates tokenizer, model backend, sampler, cache, scheduler, API, and metrics. Each component has a typed contract.

ADRs

Architecture decision records document choices. The example includes two ADRs: CPU-first and typed contracts.

Contracts

Contracts enable swappable backends and testability. The contract test verifies a backend generates expected output.

Research context

Compare runtimes by explicit subsystems: request state, scheduler, cache manager, executor, sampler, protocol, and telemetry. Pinned implementations such as DeepInfer, PagedAttention, and SGLang distinguish reusable patterns from model- or vendor-specific choices. Define invariants at subsystem boundaries before optimising.

Implementing Scheduling and Caching

Tue, 20 Oct 2026 00:00:00 +0100

Chapter 41 designed the runtime. This chapter integrates scheduling and caching.

Integration

The runtime combines admission, scheduling, caching, and cancellation. Feature flags control which components are active.

Deterministic scenarios

Deterministic concurrency scenarios allow reproducible testing. The example schedules 4 of 5 requests and cleans up cancellation.

Research context

Co-design scheduling and cache allocation; neither can be treated as an isolated queue or allocator. Make admission, preemption, prefix sharing, reclamation, and fairness observable state transitions. Prove semantic parity and isolation before claiming performance improvements. Orca introduced iteration-level scheduling; PagedAttention and SGLang extend cache and scheduling integration.

Adding Multiple Model Backends

Tue, 20 Oct 2026 00:00:00 +0100

Chapter 42 integrated scheduling. This chapter adds multiple backends.

Backend contract

A common contract allows swappable backends. The PyTorch toy backend passes conformance. vLLM and SGLang are stubs with explicit skip reasons.

Capabilities

Each backend reports capabilities. Unsupported features raise errors rather than silently falling back.

Research context

Define a backend contract for tokens, logits/sampling ownership, cache semantics, streaming events, errors, and capabilities. Use conformance tests to expose protocol and tokenizer differences. Fail closed on unsupported model/hardware/feature combinations. Transformers documentation and gpt-oss verification provide external conformance references; DeepInfer provides a pinned first-party pattern but is not independently measured.

Benchmarking Against Established Engines

Tue, 20 Oct 2026 00:00:00 +0100

Chapter 43 added backends. This chapter benchmarks fairly.

Apples-to-oranges

Comparisons must match workload, hardware, and configuration. Undocumented settings invalidate results.

What to compare

Compare output correctness, TTFT, throughput, peak memory, and task success. Report all.

Research context

Compare engines only under a shared workload, model artefact, quality target, arrival process, and cache policy. Record warm/cold state, versions, command lines, hardware, failures, and raw per-request data. Keep vendor/repository reference results in a separate table from book-reproduced measurements. MLPerf and vLLM document the metadata needed for comparability.

What We Learned: Inference From First Principles

Tue, 20 Oct 2026 00:00:00 +0100

The final chapter synthesizes what was built and what remains open.

What we learned

Inference is a system, not just a model. It spans tokenisation, computation, memory, caching, scheduling, serving, protocols, and economics.

Evidence audit

Every headline claim must map to evidence. The manifest records 45 chapters with tests and results. Failures and negative results are preserved. Audit each claim as established background, book hypothesis, local result, or first-party system report. Require an evidence chain from claim to source or raw result, environment, command, and code revision. State whether artefacts are merely available, locally functional, reproduced, or independently replicated.