Prefill/Decode Disaggregation
Split-phase workers, transport, KV movement, scheduling overhead.
Chapter 28 scaled across devices. This chapter splits the two phases of generation across different workers.
Why disaggregate
Prefill and decode have different bottlenecks. Prefill is compute-bound; decode is memory-bound. Running both on the same hardware forces a compromise. Disaggregation assigns prefill to compute-optimized workers and decode to memory-optimized workers.
Transfer cost
Disaggregation requires moving KV state from prefill workers to decode workers. If transfer cost is high, it can erase the benefit. The simulator includes a transfer-cost counterexample.
Scheduling
Separate queues for prefill and decode allow independent scaling. The scheduler must handle backpressure and avoid idle workers.
Research context
Disaggregation is not a single technique; the selected sources frame three different goals. DistServe frames it around SLO-qualified goodput, separating TTFT and TPOT objectives and making placement a scheduling problem. Splitwise frames it around heterogeneous provisioning, moving KV state between distinct hardware pools. Mooncake frames it around KV-centric storage, multi-tier caching, and early rejection.
The common cost is KV movement. The sources make transfer size, network topology, placement, and failure domains first-class concerns. Disaggregation is therefore workload-dependent: it can improve goodput when phase balance and network capacity permit, but it can also lose when transfer dominates.
Experiment
The simulator compared co-located and disaggregated regimes on four requests. Co-located total latency was 106; disaggregated was 38 with transfer cost 8. This is a simulation with simplified costs, not a production measurement.
Tests verify completion, transfer cost, and serialisation.
Limitations
- Simulation only.
- No real network transfer.
- No production scheduling.
- Simplified cost model.
- Reported gains from the cited papers are system-, workload-, and hardware-specific.
Key terms
- Disaggregation: separating phases.
- Transfer cost: moving KV state.
- Co-located: both phases on same worker.
- Goodput: useful throughput under SLOs.
- Phase splitting: distinct prefill/decode resources.
Reader exercises
- Increase transfer cost until disaggregation loses.
- Add SLO constraints.
- Vary worker counts.
- Explain when co-location is better.
References
- DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving - peer-reviewed systems paper.
- Splitwise: Efficient Generative LLM Inference Using Phase Splitting - peer-reviewed systems paper.
- Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving - peer-reviewed systems paper.
Implementation status
- Manuscript:
content/books/inference/29-chapter.md - Runnable example:
examples/ch29/ - Tests:
tests/test_ch29_disaggregation.py - Evidence:
research/ch29/sources.mdandresearch/ch29/results.json - Verification command:
python -m unittest discover -s tests -v
The chapter is a draft. It does not claim production disaggregation. The next chapter will examine reliability.