← Inference From First Principles

Prefill and Decode

A request does two different jobs. Measured on a real model: what each costs, where the textbook bottleneck fails for a small model, and how a long prompt stalls everyone else.

On this machine, GPT-2 takes 12 milliseconds to read a 16-token prompt and produce its first token, and 11 milliseconds for each token after that. Give it a prompt of 1,000 tokens and the first token takes 44 milliseconds, but the tokens after it still take 11. A 64-token reply to the short prompt spends 98 per cent of its time in the second phase. A 4-token reply to the long prompt spends 56 per cent in the first.

The same model, the same weights and the same loop do two jobs whose costs have almost nothing in common. One scales with the length of the prompt. The other does not scale with anything I could find. And when a long prompt arrives while someone else’s reply is streaming, the first job stalls the second: in my measurements, a stream producing a token every 11 milliseconds waited 56 to 73 milliseconds for its next one.

This chapter defines the two phases, measures each, and corrects a common shorthand. The shorthand is that prefill is compute-bound and decode is memory-bound. It is a good hypothesis, and for a larger model it is usually right. For GPT-2 small on this GPU, neither phase is limited by what the hypothesis says.

Where this sits

Chapter 5 showed that scoring a known sequence is parallel while choosing tokens is serial. Chapter 7 gave the roofline and the launch floor. Chapter 8 separated attention’s two shapes, many queries against many keys and one query against a cache. Prefill and decode are those two shapes applied to a request.

The Inference Lab piece added here is the request metric set: request_metrics, goodput, phase_share and a chunk scheduler, all tested without a GPU. Definitions travel with numbers, and Chapter 6’s result record gets the vocabulary it needed to say how long a request took, and to whom it was slow.

I use the library’s KV cache in this chapter, as a black box, because the two phases exist only when state is kept between steps. Chapter 10 builds the cache by hand, verifies it against the uncached loop, and measures what it saves. I do not make that comparison here.

For the Pi series, these phases are two kinds of agent traffic. An agent that feeds tool output back into the model sends a long prompt and receives a short reply, which is prefill-dominated. A chat that streams a long answer to a short question is decode-dominated. Which one a workload resembles decides what to optimise.

Two phases

Prefill processes the prompt in one forward pass of \(T\) positions. All positions are known in advance, so they run in parallel, and the pass writes the keys and values of every position into the cache. It also produces the scores for the position after the prompt, which is the first token’s distribution.

Decode then repeats a pass of one position. Each step takes the previously chosen token, runs it through every layer, reads the cache for attention, appends its own key and value, and produces the next distribution.

In terms of Chapter 8’s shapes, prefill is the \(T \times T\) problem and decode is \(1 \times T\). In terms of Chapter 7’s, prefill multiplies \(T\) tokens by every weight matrix and decode multiplies one.

The four numbers a user can perceive, and one that an operator cares about:

  • TTFT, time to first token: from the client’s request to its first received token.
  • ITL, inter-token latency: the gaps between consecutive tokens.
  • TPOT, time per output token: the average time per token after the first, \((t_{\text{last}} - t_{\text{first}})/(n-1)\).
  • End-to-end latency: from request to last token.
  • Goodput: the fraction of requests that met stated objectives for TTFT and TPOT, which is different from raw throughput.

TPOT and ITL differ. TPOT is a mean and ITL is a distribution, and a stream can have a fine TPOT and a terrible worst ITL. A unit test constructs two traces with the same TPOT, one with a single late token and one with every token slightly late, and checks that the worst gap tells them apart.

A defect in the original chapter’s example is worth recording. It defined TTFT as prefill time plus one decode step. But the prefill pass already produces the distribution for the first generated token, so TTFT is prefill plus a selection, and adding a decode step counts a token that was not needed. Its “decode” also re-ran the whole sequence at every step, so it did not distinguish decode from a second prefill.

Prefill

I measured the time to process a prompt of \(T\) tokens, with logits for the last position only, for \(T\) from 1 to 1,024. These are medians from two separate runs.

Prompt tokens GPU fp32 GPU fp16 CPU, 1 thread
1 11.5 – 12.2 ms 11.9 – 12.0 ms 40 – 45 ms
16 11.6 – 12.5 ms 13.6 – 14.0 ms 86 – 95 ms
128 11.8 – 12.5 ms 11.8 – 12.9 ms –
256 12.2 – 15.4 ms 12.4 ms 1,037 – 1,070 ms
512 22.6 – 26.3 ms 12.1 – 12.5 ms 2,096 – 2,111 ms
1,024 45.1 – 52.0 ms 16.1 – 18.3 ms –

Three regimes, one of which is not in the textbook.

A floor. On the GPU, a prompt of 1 token and a prompt of 256 cost the same, about 12 ms. That is not arithmetic and not bandwidth. It is the cost of launching GPT-2’s hundreds of small operations from Python, the effect measured in Chapter 7, plus the library’s cache bookkeeping. With a 256-fold range in arithmetic, a 12 ms floor shows that the pass is launch-bound. In fp16 the floor extends to 512 tokens, because the arithmetic is faster and takes longer to emerge from it.

Compute-bound. Beyond the floor the time grows linearly. At 1,024 tokens, fp32 processes about 20,000 to 23,000 prompt tokens per second, at 3.7 to 4.3 TFLOP/s, which is 50 to 58 per cent of the best fp32 matrix rate from Chapter 7. fp16 reaches 56,000 to 64,000 tokens per second at 10.6 to 12.0 TFLOP/s, about 35 per cent of its peak. Both are still some way from the roofline, which is a common outcome for a model this small.

One CPU thread plateaus at about 240 to 250 prompt tokens per second from 64 tokens upward, with no floor because the CPU does not pay GPU launch costs. At 4 tokens it is 71 to 86 ms.

Computing less of the output

Chapter 4 predicted that computing logits only for the last position should save about 30 per cent of the matrix arithmetic in a short prefill. The library now exposes this directly. On the CPU, where arithmetic is the whole cost, the measured time saving was 25 to 31 per cent at 64 tokens and above, against 31 per cent predicted. On the GPU it was 20 to 21 per cent at 512 and 1,024 tokens, in fp32. Below 256 tokens the saving disappears into the launch floor, as it should. This is a prediction made in Chapter 4, tested here and confirmed on the hardware where arithmetic is visible.

Decode

For the decode phase I took a cache from a prompt of 16, 128, 512 or 960 tokens, and measured 40 greedy steps with the token id read back to the host at each step, as a serving loop must. I stopped at 960 because GPT-2 has 1,024 positions; a first attempt at 1,000 prompt tokens plus 40 steps ran off the end of the position table and crashed, and I discarded it. Medians of two runs:

Context at start GPU fp32 GPU fp16 CPU, 1 thread
16 11.1 – 11.2 ms 11.0 – 12.2 ms 35 – 43 ms
128 11.1 – 11.7 ms 11.3 – 11.6 ms 39 – 42 ms
512 10.9 – 11.4 ms 11.1 – 11.4 ms 41 – 76 ms
960 10.9 – 11.4 ms 10.8 – 11.1 ms –

The step time does not depend on context between 16 and 960 tokens on the GPU. It does not depend on precision either: fp16, which halves the bytes of every weight and every cached key, is not faster than fp32. The decode step takes about 11 ms, 90 tokens per second, whatever I vary.

That rules out the textbook explanation. If decode were bandwidth-bound, halving the bytes would nearly halve the time. Computing what a bandwidth-bound step would cost makes the gap concrete. The fp32 step has to read the 475 MiB of weights plus the cache, 476 to 542 MiB. At 303 GB/s that takes 1.6 to 1.9 ms. The measured step is 11 ms. The implied bandwidth is 43 to 52 GB/s, 14 to 17 per cent of what the card can do. In fp16 it is 20 to 26 GB/s, 7 to 9 per cent.

So the step is bound by something else: the per-layer launch and bookkeeping cost of the CPU side, about 11 ms per step. It is the same floor as in prefill, and it is why a prompt of one token and a prompt of two hundred cost the same. The library’s cache adds some to it: a Chapter 7 forward pass of one token without a cache took 5.5 to 8.3 ms, and with the cache and a host readback each step takes 11. At this scale a cache does not buy a faster step. That is a measurement about this model on this GPU, and Chapter 10 treats it properly.

On the CPU the picture is different and closer to the textbook. A step costs 35 to 43 ms and rises with context (41 to 76 ms at 512 in two runs, the latter unusually noisy), because the CPU is slower than its own launch overhead and so sees the arithmetic and the cache.

What would make decode bandwidth-bound

The launch floor is fixed per layer; the weight traffic grows with model size. A model with 100 times more weight bytes has the same launch floor and 100 times more traffic. Somewhere on that scale, the traffic overtakes the floor and the step time becomes bytes divided by bandwidth. For scale, a 7-billion-parameter model in 16-bit has about 14 GB of weights, and at 303 GB/s a step cannot take less than 46 ms. That is arithmetic and not a measurement: such a model does not fit this card, and I did not run one. The point it makes is that GPT-2 small is the wrong size to demonstrate the textbook regime. It demonstrates a different regime that real small deployments, such as an on-device model, also live in.

The practical rule: find out which regime you are in before optimising. If a step takes eight times its bandwidth bound, reducing the bytes will not help. If a step takes roughly its bandwidth bound, quantising will.

Where the time goes in a request

Putting the phases together for a request on the GPU, five runs each, with last-position logits and a cached decode:

Prompt Output TTFT TPOT End to end Prefill share Decode share
16 64 12.3 ms 11.2 ms 720 ms 1.7% 98.3%
128 64 11.8 ms 11.4 ms 731 ms 1.6% 98.4%
512 64 23.1 ms 11.5 ms 747 ms 3.1% 96.9%
512 8 22.8 ms 11.4 ms 102 ms 22.1% 77.9%
1,000 16 43.6 ms 11.1 ms 211 ms 20.6% 79.4%
1,000 4 43.9 ms 11.3 ms 78 ms 56.3% 43.7%

TTFT depends on the prompt and TPOT on nothing. A request spends its time in prefill when the prompt is long and the reply short, and in decode otherwise. For a conversational workload with 100-token prompts and 300-token replies, decode is nearly all of it. For a retrieval or tool-use workload that stuffs a thousand tokens of context into every call and reads back a sentence, prefill is half or more, and TTFT, not TPOT, is what the user waits for.

Notice what TTFT does not include: tokenisation, the transfer of the prompt to the device, queueing before the request starts, and the network. In the table, TTFT is measured from the moment the prefill call began, not from the moment the client sent a request. A real TTFT is the sum of all of those and the first token’s delivery. The Lab will extend the metric to those boundaries when it has a service to measure at, in Chapter 25.

When one request’s prefill hurts another

A server that handles one request at a time has no interference to discuss. As soon as it interleaves two, the phases interact. I simulated the smallest possible case on one GPU: a request A is streaming tokens, and at its tenth step a request B arrives with a 1,000-token prompt. The loop alternates one A decode step with whatever B needs. If B’s whole prefill is one chunk, A waits for it.

Three separate runs of the same experiment, ranges shown:

B’s prefill chunk Worst gap in A’s stream Median gap in A’s stream B’s time to first token
1,000 (unchunked) 56 – 73 ms 11 – 13 ms 56 – 76 ms
512 39 – 51 ms 11 – 13 ms 73 – 91 ms
256 26 – 34 ms 11 – 13 ms 100 – 126 ms
128 26 – 34 ms 11 – 13 ms 192 – 230 ms
64 26 – 35 ms 11 – 15 ms 380 – 490 ms

Unchunked, A’s stream stalled for 56 to 73 ms, five to six times its normal 11 to 13 ms gap. The mean of A’s gaps over the forty steps would barely move: about 12 ms by a rough count, since one long gap among forty ordinary ones adds less than a millisecond to the mean. TPOT says the stream was fine. The worst ITL says a user saw the text freeze for most of a tenth of a second. A mean hides a stall; a stream’s quality is its worst gaps.

Splitting B’s prefill into chunks and interleaving one chunk per decode step bounds the stall to one chunk’s cost. At 256 tokens per chunk the worst gap fell to 26 to 34 ms. This is the idea behind chunked prefill in Sarathi-Serve, in miniature. The price is paid by B: its time to first token grew from 56 to 76 ms unchunked to 100 to 126 ms at 256 and 380 to 490 ms at 64.

The table also shows where chunking stops helping. Below 256 tokens the worst gap does not improve at all (26 to 35 ms at 64), while B’s TTFT keeps rising. A chunk of 64 tokens costs the same ~12 ms floor as a chunk of 256, so smaller chunks only multiply the fixed cost: sixteen iterations of roughly 24 to 31 ms each, which is where the 380 to 490 ms comes from. The best chunk size is the point where the chunk’s cost leaves the launch-bound floor, and it is a property of the hardware and model, not a constant.

Chunking changes the schedule and should not change the answer. The last-position logits after a chunked prefill differed from the unchunked ones by 1.5 to 1.9 \(\times 10^{-4}\), the same order as the difference between any two summation orders in float32 (Chapter 4 measured \(6 \times 10^{-5}\) to \(1.2 \times 10^{-4}\) between two implementations). It is a rounding-level difference and not a different function, and the unit tests check chunked prefill against a full pass on a small model.

Two caveats limit this experiment. It is a single-process alternation of two streams, not a serving system with queues, and the amount of interference depends on the scheduler’s policy and workload mix. And one stream of 40 steps is a small sample; the spread across my three runs, 56 to 73 ms for the same configuration, shows the GPU’s own variability.

This is the argument for keeping the phases apart in a scheduler. Chapter 15 (continuous batching) and Chapter 26 (admission and scheduling) both start from it, and Chapter 29 takes the more radical option of running the phases on different machines.

What the numbers do not show

  • GPT-2 small only. The most important finding, that neither phase is in its textbook regime, may not hold for a larger model, where weight traffic dwarfs the launch floor. The 7B figure is arithmetic and not a measurement.
  • One GPU shared with a display. Launch costs and clocks are specific to it, and the same command varied by up to 25 per cent between runs.
  • The library’s cache as a black box. Its overhead is inside the 11 ms and I have not separated it.
  • Batch size 1. Batched decode, where the weight traffic is shared and the cache traffic is not (Chapter 8), is not measured here.
  • A simulated scheduler. The interference experiment shows the mechanism and the trade-off, not what any particular server does.
  • No client or network. TTFT here begins at the model call.
  • Greedy decoding of random token prompts. The timings do not depend on prompt content, but nothing here says anything about text quality.

Inference Lab increment

The Lab now has:

  • request_metrics(request_time, token_times), returning TTFT, the list of inter-token gaps, TPOT, end-to-end time and the worst gap; goodput(metrics, ttft_slo, tpot_slo); phase_share.
  • chunk_schedule and a runnable chunked-prefill loop that is checked against an unchunked pass.
  • prefill, decode, endtoend and interference commands that produce the tables in this chapter.

The metrics module is deliberately separate from any model. Anything that can produce timestamps, whether a toy, an engine, or in Chapter 25 an HTTP client, can be scored by the same function. The rule for later chapters: report TTFT, TPOT and the worst inter-token gap, together, with the prompt and output lengths they were measured at. A single “latency” number for a language-model request is not a measurement.

What you should now be able to do

  • Define prefill and decode in terms of the shapes of attention and of the weight products.
  • Compute TTFT, TPOT, inter-token latency, end-to-end latency and goodput from a trace of timestamps.
  • Say which phase dominates a request from its prompt and output lengths.
  • Compute the bandwidth-bound floor of a decode step and compare it with a measurement to decide which regime you are in.
  • Explain why a small model on a GPU can be launch-bound in both phases.
  • Explain why a mean TPOT can hide a stall, and what chunked prefill trades for it.
  • Choose a chunk size by looking for where a chunk’s cost leaves the fixed-cost floor.

Key terms

  • Prefill: processing the prompt in one parallel pass and building the cache.
  • Decode: producing one token per pass using the cache.
  • TTFT / TPOT / ITL: time to first token; time per output token after the first; inter-token latency.
  • Goodput: throughput of requests that met their latency objectives.
  • Launch-bound: limited by the cost of issuing operations, not by arithmetic or bandwidth.
  • Chunked prefill: splitting a long prompt into pieces interleaved with other work.
  • Head-of-line blocking: a long job delaying others queued behind it.
  • Interference: one request’s work degrading another’s latency.

Reader exercises

  1. Run prefill on your hardware and find where the floor ends. How does the end of the floor move between fp32 and fp16?
  2. Compute the bandwidth-bound decode floor for a model whose weights you know, and compare with a real measurement if you can run one. Which regime is it in?
  3. Show that a trace in which one token is delayed by 50 ms and a trace in which every token is delayed by 1 ms have the same TPOT but different worst ITL. Which would a user prefer?
  4. Vary the arrival step and the stream length in interference. When does chunking stop mattering?
  5. Add a second decode stream to the interference experiment. How does each stream’s worst gap change?
  6. Measure the cached decode step with torch.cuda.synchronize() removed and the .item() readback replaced by a deferred read. How much of the 11 ms is the readback?
  7. Define a goodput objective of TTFT under 50 ms and TPOT under 15 ms. Which rows of the end-to-end table meet it?

References

Evidence in this chapter

Claim Category
Prefill is a \(T\)-position pass that builds the cache; decode is a one-position pass over it Established background
Prefill is often compute-bound and decode often bandwidth-bound Established background; shown here not to hold for GPT-2 small on this GPU
TPOT is a mean and can hide stalls Established definition, with a unit-tested example
GPU prefill is flat to 256 tokens (fp32) and linear after; 20–23k tokens/s at 1,024 Book result, two runs
Last-position logits save 25–31% on CPU and 20–21% on GPU at 512+ tokens Book result; confirms the Chapter 4 prediction
Cached decode takes about 11 ms per token on the GPU independent of context and precision Book result, two runs
The decode step is launch-bound, not bandwidth-bound Book inference from the implied bandwidth of 15–17% of the ceiling; cause not isolated
A 7B model’s decode floor is about 46 ms on this card Arithmetic, not measured
A 1,000-token prefill stalled a decoding stream for 56–73 ms Book result, three runs
Chunking bounds the stall to 26–34 ms at 256 tokens and below, at the cost of TTFT Book result, three runs
Chunked prefill matches unchunked logits to 2e-4 Book result
Batched decode, larger models, real scheduler, client latency Not measured

Implementation status

  • Manuscript: content/books/inference/09-chapter.md
  • Runnable examples: examples/ch09/prefill_decode.py (original), examples/ch09/phases.py (metrics and measurements)
  • Tests: tests/test_ch09_prefill.py, tests/test_ch09_phases.py (metrics, chunk scheduler and cache equivalence on a small GPU model; the GPU test skips without CUDA; model commands are not exercised by tests)
  • Evidence: research/ch09/sources.md, research/ch09/results.json, research/evidence-register.md
  • Verification command: python -m unittest discover -s tests -v

The chapter is a draft with draft = false. It makes no claim about larger models, batched decode, production schedulers or client-side latency. Chapter 10 builds the KV cache that makes these phases possible, verifies it against the uncached loop, and measures what it costs in memory and what it saves in time.