Inference From First Principles cover
Programmer.ie Book In development

Inference From First Principles

How a transformer becomes a request, how a request becomes tokens, and how engineering choices determine latency, throughput, cost, and correctness.

A model can know the answer; inference is the engineering that makes the answer arrive, at a cost, under constraints, and with evidence.

Inference is the missing link between a trained model and a useful system. It is not training, architecture research, prompting, or product UX. Inference owns tokenisation, tensor computation, memory, caching, batching, scheduling, quantization, distributed execution, protocol handling, and measurement.

Why this book exists

A trained transformer can be run in many ways: as a Python loop on a CPU, as an optimised GPU kernel, as a batched service, or as a distributed engine. The weights are the same; the behaviour, latency, throughput, and cost are not. The question is not only whether a model produces a token, but what computation produced it, how much memory and time it consumed, and whether the result is reproducible.

The central distinction

The book keeps these stages separate:

  1. Model parameters - the learned weights and architecture.
  2. Request - the user input, options, and constraints.
  3. Tokens - the discrete symbols the model operates on.
  4. Computation - matrix operations, attention, and feed-forward passes.
  5. Generated tokens - autoregressive output.
  6. Served response - streamed or batched API result.
  7. Measured outcome - latency, throughput, cost, and correctness under a declared workload.

A correct token is not a fast system. A fast system is not a cheap system. Each transition has to be justified and measured separately.

The cumulative experiment

The book builds one small inference system progressively. It starts with a deterministic CPU toy generator, then adds a real model, matrix execution, attention, KV caching, quantization, batching, scheduling, and finally a multi-backend runtime.

Its workloads include exact prompts, long contexts, concurrent requests, tool calls, structured outputs, and declared benchmarks. Measurements are linked to hardware, software versions, model revisions, and input manifests.

Chapter plan

The forty-five-chapter manuscript spans eight parts:

Part Chapters Focus
I. Understanding Inference 01-06 Definition, training to execution, tokens, transformer computation, generation, first engine
II. Computational Foundations 07-12 Matmul and GPU, attention cost, prefill and decode, KV cache, bandwidth, quantization
III. Making Inference Fast 13-18 FlashAttention, PagedAttention, batching, prefix cache, speculative decoding, compilation
IV. Modern Model Architectures 19-24 MoE, routing, multi-head/latent attention, sliding window, long context, architecture versus runtime
V. Building an Inference Service 25-30 API, scheduling, concurrency and streaming, distributed inference, disaggregation, reliability
VI. OpenAI, Anthropic and Applications 31-35 Protocols, API-level optimization, reasoning, tool calling, structured outputs
VII. Economics and Evaluation 36-40 Latency and throughput, cost, GPU utilization, correctness, benchmarks
VIII. Inference Lab Capstone 41-45 Runtime design, scheduling and caching, multiple backends, benchmarking, lessons

Chapters are generated individually or in explicitly approved connected batches. Drafts receive independent technical and editorial review before the author decides readiness.

Status

All 45 chapters now have manuscript, example, test-file and result-file entries. Chapters 44 and 45 have been expanded into the measured comparison and concluding synthesis. This inventory does not certify the evidence: result files retain their individual limitations and unrun work is labelled NOT_RUN. External-engine comparisons, independent reproduction and publication approval remain outstanding.

Research map

The chapter research map records exactly three selected primary sources for every planned chapter, their evidential limits, and the editorial consequences for the draft.

Method

implement → measure → explain → write → verify. CPU-first baseline, optional hardware-specific integrations. Every result must have commands, versions, workload and evidence; otherwise it is explicitly NOT_RUN.

Contents

Chapters

01

What Is Inference?

Distinguish mathematical inference, LLM forward passes, token generation, and production serving, and measure where one request actually spends its time.

02

From Training to Execution

What training leaves behind, what must be true before it is executed, and what the first request pays that later ones do not.

03

Tokens, Embeddings and Probability

From text to token ids to a vector per token to scores over the vocabulary, and the rules that turn scores into a choice. Measured on a real model.

04

The Transformer as a Computation

A GPT-2 forward pass written out by hand, checked against the library, and used to count parameters, FLOPs and time component by component.

05

Autoregressive Generation

A forward pass becomes a sequence: the cost of the loop, why checking is cheaper than producing, and how to stop and stream without corrupting the text.

06

Building Our First Inference Engine

Assemble tokeniser, backend, sampler, loop and record into one engine with a stable interface, check each replaceable part against a reference, and measure the baseline.

08

Attention and Its Computational Cost

Attention is quadratic in arithmetic, but what hurts first is the score tensor. Measured memory, time and bandwidth for prefill-shaped and decode-shaped attention.

09

Prefill and Decode

A request does two different jobs. Measured on a real model: what each costs, where the textbook bottleneck fails for a small model, and how a long prompt stalls everyone else.

10

Key-Value Caching

A KV cache written by hand into GPT-2: why it is exact, what it saves, what it costs in memory, and how it fails when its state is wrong.

12

Quantization and Numerical Precision

Four different things are called quantisation quality: storage, reconstruction error, model behaviour and speed. Measured separately on GPT-2, they disagree.

13

FlashAttention and Optimized Kernels

Block-wise exact attention is an algorithm; making it fast is a kernel. Both measured, plus an honest account of what could not be run here.

14

PagedAttention

A KV cache in fixed-size blocks: how much memory contiguous allocation wastes, what block tables, reference counts and copy-on-write buy, and what greedy admission costs.

15

Continuous Batching

A real GPT-2 server under static and continuous scheduling: latency, goodput and wasted work at four offered loads, plus what chunked prefill and queue policy do to fairness.

16

Prefix Caching

Correct prefix reuse, its prefill savings, capacity and isolation trade-offs, and a measured timing side channel in a scoped block cache.

17

Speculative Decoding

Letting a small model guess and a large one check: the acceptance rules that keep the output exact, what the guesses are worth on real text, and where the arithmetic says it should and should not pay.

18

Compilation and Execution Graphs

Capturing a decode step as a graph: what Dynamo's guards promise, what a CUDA graph removes, what an Inductor-compiled kernel fuses, what each costs to build, and why the first claim in this chapter's earlier draft was wrong.

19

Mixture-of-Experts Inference

Storing many experts and computing with a few: what a batch of tokens makes a layer read, four ways to run the layer, what each costs on a GPU, and why capacity limits make a token's answer depend on its neighbours.

20

Expert Routing and Load Balancing

What a trained router actually does to a serving system: how three balancing methods differ, what a capacity limit costs in quality on learned routing, how placement from measured load cuts the straggler, and what communication would add.

21

Multi-Head and Multi-Latent Attention

How many keys and values a model keeps decides its cache, its decode bandwidth and how many requests fit: real cache sizes from cached models, why a naive grouped-query kernel saves nothing, what converting a multi-head model costs, and a latent-attention layer built from the paper and measured.

22

Sliding-Window Attention

A window bounds what attention reads, but not what a model needs: a rolling cache measured on Gemma-2, how far influence travels through stacked windows, what a plain window does to a model not trained for one, and why a masked window is not yet a speed-up.

23

Long-Context Inference

A context length is five claims, not one: that the model accepts the tokens, behaves at those positions, fits in memory, answers in time, and uses what it is given. Each measured on a model that advertises 128K, on a card that can hold a quarter of it.

24

Model Architecture Versus Runtime Architecture

Changing one thing at a time on a real model to find where the model ends and the runtime begins, what the software stack does when asked for something it cannot do, and an admission function that refuses before any work is done.

25

From Inference Loop to API

Putting the engine behind an HTTP interface and measuring what the boundary has to decide: where a request's time goes, how a character limit fails as a token budget, which errors a client can act on, how a schema may change, and what a shared backend does to concurrent callers.

26

Request Scheduling and Admission Control

What a queue in front of a one-at-a-time engine does when traffic exceeds its capacity, measured on a live service and a simulator checked against queueing theory: four admission policies, the deadline that makes an answer worth having, who gets turned away, and what stale estimates cost.

27

Concurrent Requests and Streaming

Thirty-two streams on one batched engine over real sockets: what overlap buys, why a stream's first token waits for its neighbours, what one slow reader does under four policies, and how long the engine goes on working for a client that has gone.

28

Distributed Inference

GPT-2 split across processes by tensor and pipeline parallelism, exact to the last token and no faster, and the cost model that explains why: what each strategy splits, what it must communicate, what that communication costs, and what a machine has to provide before splitting helps.

29

Prefill/Decode Disaggregation

A real prefill process and a real decode process joined by a socket, exact to the token and worse than one engine on a single shared GPU; the simulator that explains why, and what it says about the hardware, the workload and the prefill-to-decode ratio that would make the split pay.

30

Reliability, Cancellation and Recovery

Faults injected at every transition of a request's life in the Chapter 27 engine, what leaked, what hung and who was told what; an engine that rolls each one back; cancellation in every state; inputs that cannot be served; a worker that is killed, hung or hit by a device-side assert; and what retries do to a service past its capacity.

31

Inference Protocols and Harmony

The Harmony message format as a boundary between what an application controls and what it does not: a renderer and a streaming parser checked token for token against the reference library, hostile text through each place it can enter a prompt, tool calls decided by stop tokens and structure, and what the framing costs in tokens and in reusable cache.

32

Claude, Caching and API-Level Optimization

What a provider's prompt-caching contract lets an application see, and what it does not: a mock that implements only the documented rules, two different internals that the contract cannot tell apart, break-even request rates for the two lifetimes, workloads priced at dated published prices, and an opt-in harness that refuses to spend money unless it is told to.

33

Reasoning Models and Inference-Time Compute

A 0.6-billion-parameter model that can think, on 200 maths problems: accuracy against thinking budget, against parallel samples, and against an adaptive rule, with the tokens, the waiting time and the cost per problem solved; and the provider-side controls, versioned and untried.

34

Tool Calling and Agentic Inference

A tool loop around a real 0.6-billion-parameter model, built and measured end to end: a calculator that cannot be talked into running code, a renderer that cannot be forged, the price of resuming after a tool result, and the surprising answer to whether the tool made the model better.

35

Structured Outputs and Constrained Decoding

A schema compiled to a finite-state constraint over a real tokeniser, checked against the jsonschema library, Python's re and a reference implementation, and run on a 0.6-billion-parameter model: what a guaranteed-valid output costs, what it does not guarantee, and where it breaks.

36

Latency, Throughput and Time to First Token

A metrics collector and report renderer built on timestamp pairs, checked against numpy, two clocks and a queueing formula, and used on a small real server under open- and closed-loop load: what the clocks, the percentiles, the denominators and the load generator each do to the numbers.

37

Cost per Million Tokens

A configurable cost estimator whose every input says where it came from, fed with measured power and throughput and with dated prices: what cost per token is made of, how utilisation and batch size move it, how a forecast differs from a metered run and from a bill, and why the price of a token is not the price of an answer.

38

GPU Utilization and Capacity Planning

What the GPU's utilisation counter says against what the work used, where memory limits turn out to be performance cliffs, and a capacity model built from load sweeps with confidence bands, repeated runs and a refusal to extrapolate.

39

Correctness, Determinism and Reproducibility

A cross-run comparison harness with a manifest, tested on a real model: repeatable bit for bit but not invariant to shape, what exact equality and numeric tolerance each say, how far a changed batch moves the tokens, and what it takes to make a seed mean the same thing in every batch.

40

Designing Meaningful Inference Benchmarks

A benchmark suite with a frozen workload manifest, full configurations, interleaved paired trials and machine-readable results, run on four configurations of a small server: what repeated trials, an A/A comparison and a choice of unit of replication do to the verdicts.

41

Designing the Inference Runtime

The capstone runtime specified before it is optimised: typed contracts, nine decision records tied to forty-seven checks, a dependency table enforced by reading the imports, and thirty-nine deliberately faulty adapters used to find out whether the checks can fail.

42

Implementing Scheduling and Caching

A scheduler, the engine's own books for cache memory and a copying prefix cache added to the capstone behind two flags: what they change on a real model when the compute is still serial, and a bounded cancellation enumeration over 368 real-model runs, with explicit limits on what it proves.

43

Adding Multiple Model Backends

A batched-decode capability, a second level of backend for engines that own sampling, and a conformance matrix that says pass, fail or skip-with-a-reason: what the same scheduler does with a backend that decodes sixteen sequences in one call, and what a backend that cannot run looks like.

44

Benchmarking Against Established Engines

A matched comparison of the capstone and library decode loops: first-token latency, throughput, memory, token agreement and extraction success, with explicit limits on comparisons that could not run.

45

What We Learned: Inference From First Principles

The completed argument of the book: preserve identity through every inference stage, explain gains with mechanisms, audit claims against artefacts, and choose the next experiment from what remains uncertain.