Distributed Inference
GPT-2 split across processes by tensor and pipeline parallelism, exact to the last token and no faster, and the cost model that explains why: what each strategy splits, what it must communicate, what that communication costs, and what a machine has to provide before splitting helps.
GPT-2 was partitioned across up to six CPU processes: at six ranks, each process computed one sixth of each layer’s attention heads and feed-forward intermediate width. In fifteen tensor-parallel runs across one, two, three, four and six ranks, all 25 generated tokens matched the Hugging Face reference on the tested prompt; the largest difference in the final prompt logits was 9.2 × 10⁻⁵. Partitioned computation did not make decoding faster: the measured median step took 23.1–23.9 ms at one rank and 54.4–58.7 ms at six. This is a demonstration of arithmetic partitioning, not a memory-residency demonstration: the laboratory workers load a full GPT-2 checkpoint before constructing their partitioned tensors, and have not been designed or measured to avoid that temporary and potentially continuing replicated residency.
The output tokens matched, but additional CPU processes did not supply additional independent memory bandwidth. In a separate test, eight full-model replicas with no collectives achieved 37.8–44.7 generated tokens/s in aggregate, against 39.6–40.6 for one replica. A one-rank decode step was estimated to stream 497.8 MB of weights and achieved 20.8–21.5 GB/s by that calculation; a separate multi-process memory-reading benchmark plateaued around 25 GB/s, with substantial run-to-run variation. Those observations support a shared-bandwidth bottleneck on this CPU, rather than proving that communication is the only cause of the tensor-parallel slowdown.
The central lesson is conditional: splitting inference helps capacity when partitions truly reside on separate devices, and it can help token latency when each device contributes enough additional compute or memory bandwidth to repay communication. This machine has only one GPU; its real partitioning experiments run on CPU processes sharing memory. Accordingly, the chapter distinguishes a measured CPU mechanism, a link-cost model, and hypothetical multi-device projections. None of the projections is presented as an observed multi-GPU speedup or proof that a particular deployment will fit.
The question
This test machine has one GPU, no available NCCL backend in the installed Windows build, and 24 logical CPU threads. It therefore cannot measure distributed GPU execution, NVLink, or an inter-node inference service. The starter instead counted bytes in a toy simulator and described that limit. This chapter has three kinds of material and keeps them apart.
First, a cost model: what each strategy splits, how many bytes and messages per generated token it needs, and what they cost on a link described by a latency and a bandwidth. Second, real distributed execution: GPT-2 split by heads and by layers across operating-system processes that talk through PyTorch’s gloo backend, checked for exactness and timed, with the collectives measured on their own. Every CPU worker starts by loading the full reference checkpoint, so these runs do not establish sharded process memory consumption. Third, records of what was not run: NCCL, a second GPU, a second node, and a load-only-the-shard deployment.
The questions are four. Does this tested split preserve greedy outputs and how close are its logits? What does the communication cost, and does the standard two-constant model predict it? When does splitting make a token faster, and what did it do here? And, for real model configurations, which strategies might fit on which devices under explicitly incomplete memory accounting and how much latency per collective can a split afford?
Where this sits
Matrix Multiplication and GPU Execution measured about 303 GB/s in a read-dominated benchmark on this GPU. This chapter uses that figure as a hypothetical per-device bandwidth in its multi-GPU model, not as the bandwidth observed by the CPU process experiments; their independent measured memory-streaming rates are roughly an order of magnitude lower. A GPU microbenchmark ceiling is not necessarily sustainable model-serving bandwidth. PagedAttention, Multi-Head and Multi-Latent Attention and Long-Context Inference gave the size of the KV cache per token and showed that grouped-query attention shrinks it; here the number of KV heads also limits how far attention can be split. Model Architecture Versus Runtime Architecture read ten configurations into fingerprints and said every option that changes what is computed belongs in the manifest; the parallel layout (which strategy, over how many devices) will have to join them, and this chapter checks that question for GPT-2 greedy token sequences, not for every checkpoint, precision or prompt. Concurrent Requests and Streaming ran many streams on one engine; the engine here is smaller and its purpose is different.
The chapter corrects the starter’s idealised communication accounting; the replacement formulas themselves assume specific collective algorithms and count bytes per rank, not a measured backend implementation. Its simulator charged tensor parallelism (p − 1) times the activation bytes, called data parallelism’s cost “gradient/activation all-reduce” (inference has no gradients, as the starter’s own text says), and treated the other strategies as one activation each. None of those is what a collective moves. A test in the repository compares the starter’s tensor figure of 300 bytes for four devices with the 150 an ideal ring all-reduce transfers per participant for the same buffer.
Prefill/Decode Disaggregation takes up splitting prefill from decode across machines. That move transfers live KV state, not just the small layer-boundary activations considered here, and therefore requires a separate accounting of cache bytes, transfers and ownership. Reliability, Cancellation and Recovery takes up what happens when one of several devices fails.
What each strategy splits
Data parallelism puts a whole copy of the model on each device and gives each request to one of them. Nothing is split and nothing is synchronised: unlike training there are no gradients. What remains is routing, the manifest of From Training to Execution, and the choice of replica. It can add serving capacity without adding intra-request collectives, provided replicas have independent enough resources and the router does not add meaningful delay. On shared memory, replication need not add aggregate throughput; the CPU experiment later in this chapter is a direct counterexample to any unconditional scaling claim. Every replica also needs its own usable model and KV-memory budget.
Tensor parallelism cuts each layer across devices. The scheme is Megatron-LM’s. The first matrix of the feed-forward block is split by columns, so each device applies the activation function to its own slice with no communication; the second is split by rows, so the devices’ partial outputs add up to the layer’s output. Attention is split by heads: the query, key and value projections by columns, so a head’s whole computation is local, and the output projection by rows. The paper counts the result as two all-reduces in the forward pass of a layer (one after attention, one after the feed-forward), and two more in the backward pass, which inference does not have. Megatron-LM also discusses vocabulary-parallel embeddings and output logits. This lab specifically shards the output head by vocabulary and gathers its logits, but retains the input embedding in each worker; its implementation is not a complete memory-sharded Megatron deployment. The final gather is one choice for greedy inference, not a universal requirement: distributed top-k or argmax reductions can avoid reconstructing all vocabulary logits when the API needs only the chosen token.
For the specific laboratory implementation and a single decoded token at more than one tensor-parallel rank, GPT-2 makes 2 × 12 = 24 all-reduces of 768 float32 values (3,072 bytes of input per rank each), followed by one vocabulary-logit all-gather (roughly 100 KB contributed per rank at two ranks). A one-rank run performs no network collectives; larger decode batches scale the activation-buffer and logit-buffer sizes. The cost model assigns Llama-3.1-8B 64 all-reduces of 8 KiB activations and one final gather for a single-token, batch-one forward pass. This is an assumed Megatron-style layout, not an observed Llama execution. The number of layer reductions is set by depth, whereas their collective latency and topology-dependent algorithm change with the device count.
Pipeline parallelism assigns consecutive model layers to stages and passes activations between them; a final token choice or its information must make it back to the first stage for the next autoregressive step. One sequence therefore traverses the stages serially. Independent sequences can occupy different stages simultaneously, which improves throughput without reducing the path length of one token. GPipe’s familiar bubble term, \((K-1)/(M+K-1)\), describes a fill-and-drain schedule of \(M\) independent micro-batches over \(K\) equally timed stages in training. A full forward wave of \(m\) independent decode inputs likewise needs \(m+p-1\) stage intervals in the elementary equal-stage, no-overlap model, not merely \(\max(m,p)\). The laboratory’s pipeline_round_time helper currently returns max(sequences, stages) * stage_s; that is an optimistic occupancy proxy, not a validated fill-and-drain completion-time formula. Its derived pipeline-throughput estimates must be recalculated after correcting the helper and specifying how consecutive autoregressive waves overlap. The measured multi-process pipeline timings later in the chapter are separate observations and do not depend on treating that helper as exact.
Expert parallelism places the experts of a mixture-of-experts layer on different devices and sends each token to the devices that own the experts it chose, and back: typically a dispatch and a return all-to-all for a routed expert computation. The number of message rounds, peer volume and imbalance depend on the routing layout; the lab times the data exchange, not expert compute, overlap or load balancing. Mixture-of-Experts Inference and Expert Routing and Load Balancing built the dispatch and its balancing; here only synthetic dispatch communication is measured.
The DeepInfer README at the commit pinned in this book lists, among its supported features, expert parallelism with load balancing (“MoE EP + EPLB”), data parallelism for attention, and context parallelism. The absence of tensor and pipeline parallelism from that feature list is not evidence that such techniques are absent from every component or revision. Its benchmark section reports 4 GPUs for prefill and 64 for decode, each within one scale-up domain, without naming the interconnect or stating a maximum number of devices, and lists a KV cache system, prefill/decode disaggregation and load balancing across replicas as required for a complete service and not included. That is the authors’ description of a system on hardware I do not have; I did not run it, and those benchmark device counts are not a measured configuration for this chapter.
A cost model
The alpha–beta (Hockney) approximation models one transfer of \(n\) bytes as \(\alpha+n/\beta\) seconds, where \(\alpha\) is a startup latency and \(\beta\) is an effective bandwidth. It excludes contention, software scheduling, algorithm switches and overlap, and its parameters belong to a specified operation and transport. In the ideal ring algorithm, an all-reduce over \(p\) devices is a reduce-scatter followed by an all-gather, each \(p − 1\) steps, so
For decode the model needs two more things. The model treats batch-one decode as bandwidth dominated, so its lower-bound-like estimate divides idealised streamed-weight and KV bytes by an assumed per-device bandwidth; it omits compute ceilings, memory hierarchy, synchronisation, launches and allocator overhead. Those bytes are not the bytes the model occupies: the input embedding is stored but a step reads one row of it, while the output head is read in full. The cost model keeps stored bytes, which decide whether a device can hold its share, apart from streamed bytes, which decide how long a step takes. For tensor parallelism it adds modeled collective time to that device-memory term. For pipeline parallelism, individual stage occupancy and the full sequence path are distinct quantities; the planner approximates them separately.
The model needs the shape of a model, and the formula that gives it should be exact or its predictions are suspect. I checked it against the headers of the checkpoints in the local cache. For seven cached models, after the stated checkpoint-specific counting corrections, its parameter totals matched the parameter-shaped values counted from the local safetensors headers: GPT-2 (124,439,808), TinyLlama (1,100,048,384), Phi-3-mini (3,821,079,552), Gemma-2-2B (2,614,341,888), Mistral-7B (7,248,023,552), Llama-3.2-1B (1,235,814,400) and Llama-3.2-3B (3,212,749,824). Qwen3-1.7B (1,720,574,976) and Llama-3.1-8B (8,030,261,248) remain formula-only because their weights were not cached; Pythia is not covered by this particular shape formula. Exact agreement in parameter counts does not verify tensor names, values, parallel padding, memory residency or execution correctness. The first comparison disagreed for three models, and each disagreement was a fact about the checkpoint and not an error of arithmetic. GPT-2’s file also stores its causal-mask buffers, 12 × 1,024² = 12,582,912 values that are not parameters. Mistral’s folder holds a second copy of the weights. And Gemma-2 ties its input and output embeddings by default without its configuration saying so, and has four normalisation layers per block where the Llama layout has two. A formula checked on Llama alone would have missed these conventions. The later partition-memory estimates are a different model and were not validated against allocated memory on multiple GPUs.
What this machine can do
Before the experiments, what is and is not available, recorded by the gates check or the stated probe rather than inferred from an installed CUDA wheel:
| Capability | Here |
|---|---|
torch.distributed with the gloo backend (CPU, over TCP) |
available, after setting USE_LIBUV=0 (the Windows build has no libuv) |
| NCCL | not available in this build; the PyTorch documentation lists NCCL as supporting GPU collectives and Windows as supporting every backend but NCCL |
| MPI | not available |
| CUDA devices | one |
gloo reduce_scatter |
not supported (it raised), as the documentation’s table says |
gloo all_to_all_single |
succeeded in this local version-specific probe; the documentation table describes a different API entry and should not be overgeneralised |
So NCCL, multi-GPU and multi-node runs are NOT_RUN. The distributed work that does run uses CPU tensors, gloo and operating-system processes. Individual tests check backend and CPU availability and may skip when their prerequisites are absent; their passing on this machine is not evidence that they ran in every CI environment. No test has verified genuine shard-only loading, rank failure recovery or GPU collectives.
Does splitting change the answer?
Hypothesis (book hypothesis). Splitting GPT-2 by heads and feed-forward width, or by layers, preserves the tested greedy token choices despite a changed reduction order and potentially different floating-point logits. This is a falsifiable, prompt-specific hypothesis—not a theorem of bitwise model equivalence.
Test. The lab implements GPT-2’s forward pass in a form that can hold any share of the heads and of the feed-forward width, and any range of layers, with a KV cache for its assigned heads (GPT2Part). Every worker begins by loading the reference model and then constructing its part; this experiment does not qualify a reduced-memory loader. It is compared with the Hugging Face model on the chapter’s selected 32-token book prompt and 25 greedy output tokens; the tests use additional small synthetic configurations, but there is no broad prompt suite for the measured 124M model. Tensor parallelism ran on 1, 2, 3, 4 and 6 processes, three times each. Pipeline parallelism ran on 1, 2 and 4 stages with 1, 2, 4 and 8 sequences in flight, three times each.
Result. In 15 of 15 tensor-parallel runs the 25 tokens equalled the library’s, every rank agreed, and the largest difference in the final logits of the prompt was 9.2e-5. In every pipeline-parallel run every sequence equalled the library’s greedy tokens. The hypothesis was not falsified on this GPT-2 float32 greedy workload. Reduction order differs (the logits are not bitwise equal, and Speculative Decoding and Model Architecture Versus Runtime Architecture explain what a near-tie does with that), so matching output token ids is not proof of identical logits or identical behaviour across inputs; half precision, other models and sampled decoding were not run.
With two or more ranks and the vocabulary-parallel output head enabled, the measured path performs 24 all-reduces and one all-gather per decode step. Its single-rank baseline performs no inter-process collectives. A different logits-selection strategy or replicated head would change the final collective and memory traffic.
What the collectives cost
Collectives were microbenchmarked with 2, 4 and 8 CPU ranks communicating through gloo over loopback TCP, using buffers from four bytes to 16 MiB. The times measure the installed collective implementations and operating-system scheduling—not a GPU fabric or a proven ring algorithm.
| Observed median time across 64 B–64 KiB inputs | 2 processes | 4 processes | 8 processes |
|---|---|---|---|
| all-reduce | 132 to 140 µs | 339 to 362 µs | 1.59 to 1.68 ms |
| all-gather | 90 to 111 µs | 206 to 246 µs | 0.88 to 1.13 ms |
| all-to-all | 57 to 73 µs | 104 to 157 µs | 375 to 599 µs |
| ring send/receive | 52 to 65 µs | 57 to 78 µs | 117 to 145 µs |
Across the tested 64 B–64 KiB range, fixed per-operation and software costs dominated the input-size effect on this gloo transport; the table reports ranges across sizes, not one constant. The two-process all-reduce ranged from 132 to 140 µs, but at eight ranks it ranged from 1.59 to 1.68 ms. That is the regime of decode. The all-reduce of a token’s activation is 3 KB. For GPT-2’s 3,072-byte activation reduction in this particular microbenchmark, latency dominates; the bandwidth term is not literally zero. The all-gather of vocabulary logits is a different and larger payload, and cannot be discarded from decode cost.
Does the two-constant model predict the latency? Using the two-process all-reduce fit (118.7 µs intercept, corresponding to \(\alpha \approx 59\) µs under the assumed ring model) the model predicts a 4-process all-reduce of 356 µs; the measured flat value is 339 to 362. It predicts 830 µs for 8 processes; the measured value is 1.59 to 1.68 ms, twice as much. The all-to-all follows the model more closely: with \(\alpha\) taken from the 54 µs of a point-to-point message, \((p−1)\alpha\) gives 162 µs for 4 processes (measured 104 to 157) and 378 µs for 8 (measured 375 to 599). Thus a two-rank fitted constant reasonably interpolates this four-rank case but underpredicts eight ranks by about a factor of two. The measurements do not identify the cause: collective algorithm, process scheduling, resource contention and TCP behaviour were not isolated. The fit is transport- and process-count-specific, not a general network latency constant.
It fails for large messages on this transport. The fitted model underpredicted held-out all-reduces of 4 MB and 16 MB by 86% and 94% at 2 processes, by 65% and 81% at 4, and by 30% and 74% at 8. A finer grid made the failure specific. At 2 processes, the all-reduce took 0.19 to 0.20 ms at 256 KB, 0.41 to 0.52 at 1 MB, then 11.7 to 14.3 ms at 2 MB, and at 4 MB 3.2, 14.3 and 11.9 ms in three repetitions: a jump of about twenty-five times for twice the bytes, and then no regularity. The 4-process times were smooth (1.12 to 1.20 ms at 2 MB, 5.0 to 5.3 at 4 MB). A possible connection to Windows scheduling or collective implementation thresholds is an untested hypothesis; no scheduler tick was actually measured. For the tested GPT-2 batch-one decode this large-payload discontinuity was not exercised, although larger batches, vocabulary handling or other model architectures may reach different sizes. For a prefill that all-reduces a long prompt’s activations it would, and the simple extrapolation should not be trusted on this transport without a size- and rank-specific remeasurement.
Splitting the model, and what it did to the step
Tensor-parallel GPT-2, decode steps of a 25-token generation after a 32-token prompt, three runs per size, the recorded range of median step times across three runs and time spent in collective calls. The separate step-minus-collective quantity is arithmetic subtraction, not an independently instrumented compute-only pass:
| Processes | Step | Time in collectives | Step minus collectives |
|---|---|---|---|
| 1 | 23.1 to 23.9 ms | 0 | 23.1 to 23.9 ms |
| 2 | 25.7 to 30.5 | 4.4 to 5.3 | 21.3 to 25.1 |
| 3 | 32.7 to 45.3 | 8.6 to 16.1 | 24.1 to 29.2 |
| 4 | 32.8 to 47.5 | 11.6 to 21.7 | 21.1 to 25.8 |
| 6 | 54.4 to 58.7 | 32.6 to 35.2 | 21.8 to 24.3 |
Two things are visible. Time measured inside collective calls grew with the number of processes: 25 collectives per step cost about 180 to 210 µs each at 2 processes, 460 to 870 µs at 4 and 1.3 to 1.4 ms at 6. The residual step time minus collective-call time did not materially fall. Each rank computed on a fraction of the layer matrices, yet its residual step time resembled the one-rank time. Because every worker also loaded the full checkpoint and the subtraction includes other work and scheduling effects, this is evidence about the execution path, not a verified per-rank memory footprint or pure compute time.
The communication is partly predicted. From the microbenchmarks, 24 all-reduces at their 1 KB time plus one all-gather at 64 KB come to 3.3 ms at 2 processes and 8.4 ms at 4, against 4.4 to 5.3 and 11.6 to 21.7 measured. The timed call includes synchronisation and skew between ranks that the isolated-collective estimate may not represent. Consistent with that, observed collective-call time exceeded this simple sum by 35 to 63% at 2 processes and by 37 to 157% at 4. As a prediction of the order of magnitude and the trend it is useful; as a deployment estimate it needs matched-size benchmarking and explicit treatment of rank skew.
Where the head goes matters. With the output head replicated on all four processes instead of split by vocabulary, a step took 64.5 to 69.9 ms against 45.7 to 50.1 ms with the head split. The head is 154 MB of float32 weights, a third of the model, and reading replicated output-head weights on each of four ranks increases aggregate memory traffic relative to the vocabulary-partitioned head. The per-rank traffic and operating-system cache effects were not measured separately; the four-rank comparison supports the direction, not a precisely established fourfold traffic increase. That is the same lesson as the main one, arrived at from a different side.
Prefill behaves differently, and the difference is informative. A 32-token prefill took 73 to 76 ms on one process and 48 to 65 ms on two and 40 to 64 ms on four, an improvement in some runs (73–76 ms at one rank versus a best observed 40 ms at four ranks, about 1.8×, with overlapping ranges), before it got worse at six (76 to 98 ms). Processing 32 prompt tokens together changes arithmetic intensity and the relative importance of weight streaming. The experiment did not profile CPU arithmetic utilisation or memory bandwidth during prefill, so an arithmetic-bound explanation remains plausible rather than demonstrated.
Why: the memory system is shared
Hypothesis (book hypothesis). The decode step on this machine is limited by memory bandwidth that all the processes share, so partitioning among these processes yields little or no reduction in residual decode time while adding collectives. This conclusion is specific to this shared-memory CPU and its implementation, not a universal limit on distributed inference.
Two measurements test it independently of the tensor-parallel runs.
Independent replicas. Each process decoded its own sequence, with nothing to communicate, and all started together. Aggregate tokens per second over three runs: 1 replica, 39.6 to 40.6; 2 replicas, 37.9 to 50.1; 4, 38.9 to 43.4; 6, 44.6 to 50.0; 8, 37.8 to 44.7. Increasing replica count did not produce proportional aggregate scaling. Across runs, the one-replica total was 39.6–40.6 tokens/s and the eight-replica total 37.8–44.7. Individual replica rates declined as the aggregate work competed for shared resources; the intermediate counts have overlapping and sometimes higher aggregate results. A decode step streams 497.8 MB of weights, and the resulting implied throughput was 20.8–21.5 GB/s. This is an estimated weight-read rate from modeled bytes over wall time, not a hardware counter recording physical DRAM traffic.
A memory benchmark. Each process summed its own 512 MiB tensor, simultaneously. The aggregate effective tensor-read rate for simultaneous summations across 2, 4, 6, 8 and 12 processes was 16 to 24, 21 to 26, 19 to 28, 25 to 30 and 22 to 27 GB/s: a broad plateau around 25 GB/s, peaking at approximately 30 GB/s in these tests, with less bandwidth available on average per reader as their number increased. CPU cache effects, NUMA placement and true DRAM transactions were not separately instrumented.
The two models predict different things. Under an idealised independent-bandwidth model, perfectly sharded weight reads could reduce the 23 ms baseline toward 12, 6 and 4 ms at 2, 4 and 6 devices, before other work. If the bandwidth is shared, it would stay at about 23 ms. The measured values were 21 to 29 at every size. The shared-resource hypothesis is supported on this CPU setup, and the explanation is not circular: it is cross-checked by a separate memory-reading workload and on a prediction, the constant non-communication time, that could have differed. It does not establish which share of the remaining time is DRAM traffic, CPU scheduling, full-checkpoint replication or framework overhead.
That is also why a pure-CPU experiment of this kind validates the mechanism (the tested greedy outputs match; the collectives have the measured gloo costs) and not the speed-up. A genuine multi-GPU partition can draw on separate device memories and memory channels, making a different result possible—subject to interconnect, compute and memory placement. The lab neither loads only shards into GPU memories nor measures the associated gain.
The pipeline hits the same ceiling
Pipeline-parallel generation of 16 tokens from 32-token prompts, aggregate generated tokens per second for complete runs, including prefill, across three repetitions. These are measured worker results rather than predictions from the pipeline_round_time helper:
| Stages | 1 sequence | 2 sequences | 4 sequences | 8 sequences |
|---|---|---|---|---|
| 1 | 29.5 to 36.1 | 27.3 to 33.0 | 33.6 to 36.0 | 32.8 to 33.2 |
| 2 | 32.9 to 34.9 | 38.6 to 41.2 | 34.7 to 40.7 | 39.5 to 41.7 |
| 4 | 24.9 to 32.3 | 28.6 to 39.1 | 33.5 to 38.7 | 38.2 to 42.4 |
The model’s prediction for one sequence holds: with one sequence in flight, 2 or 4 stages are no faster than one stage (it visits each in turn). With more independent sequences in flight, some two- and four-stage configurations reach roughly 40–42 tokens/s; their improvement over the approximately 33-token/s one-stage, eight-sequence case is modest and varies by run. The measured table does not establish a general pipeline scaling factor or validate the planner’s simplified max(m, p) round-time formula. Competition for shared CPU memory is consistent with the replica experiment but was not measured concurrently with each pipeline cell. A pipeline of 4 stages with 8 sequences in flight is the best case for the strategy on paper, and shared-memory contention is a strong explanation rather than a uniquely identified causal limit.
Expert dispatch
A synthetic all-to-all proxy for moving routed activations toward experts and back was measured for 768-wide activations, top-2 routing, spread over 2, 4 and 8 processes. For a decode batch of 1 or 8 tokens it took 65 µs at 2 processes, 106 to 121 µs at 4 and 370 to 385 µs at 8: all latency. A straightforward, non-overlapped expert-parallel layer would invoke a dispatch and a return exchange; actual MoE implementations may fuse, overlap or otherwise alter that pattern. At 64 and 512 tokens the times rose to 0.17 to 0.52 ms and 3.6 to 5.5 ms at 4 and 8 processes. At 2 processes the same erratic jumps appeared as in the all-reduce (14.8 ms for 64 tokens, then 1.2 ms for 512), which the underlying results.json retains. This chapter does not measure routing skew, expert execution or token completion latency of a full distributed MoE.
What a link costs on this machine
The machine has one GPU, but it has a link: the host to device copy. With pinned memory, a copy of 8 KB took 11.1 µs to the device and 9.2 µs back; 1 MB took 95 and 88 µs (11.0 and 11.9 GB/s); 16 MB took 1.35 and 1.30 ms (12.5 and 12.9 GB/s); 256 MB took 21.8 and 21.7 ms (12.3 and 12.4 GB/s). These pinned host-to-device and device-to-host measurements are consistent with roughly 10 µs small-message overhead and about 12.4 GB/s sustained copy rate for the larger measured buffers. A two-GPU transfer routed through host memory might require two such legs, but PCIe topology, peer access, copy engines and concurrency were not tested; doubling the measurement is a hypothetical link scenario, not a measured GPU-to-GPU transfer.
Modelled for hardware this machine does not have
The model can say what the measured constants imply for real configurations, and it has to be clear that this part is arithmetic and not measurement. The setting: decode of one sequence at a 4,096-token context in fp16, assuming each hypothetical GPU independently sustains 303 GB/s, the best observed read-dominated Matrix Multiplication and GPU Execution figure. This is not an achieved per-device transformer decode rate; the model sums estimated memory service and serial communication without execution overlap. Two are measured here: a gloo loopback point-to-point proxy (54 µs, 1.2 GB/s, not a GPU fabric) and a two-leg host-staging proxy (20 µs, 6.2 GB/s, inferred by doubling measured PCIe copy costs). The third uses the bandwidth the Megatron-LM paper reports for NVSwitch inside a server, 300 GB/s in the authors’ hardware description, with an assumed 10 µs collective/startup latency. No real GPU collective performance was measured for any of these scenarios.
Llama-3.1-8B in fp16 needs 14.96 GiB for its weights, which does not fit a 12 GiB card. It therefore cannot fit entirely as fp16 weights on this particular 12 GiB GPU. Partitioning across at least two devices is a plausible capacity route if the shards are really loaded separately and room remains for KV cache, scratch buffers, allocator fragmentation, kernels and serving runtime. Neither the model’s fits_12GiB check nor the CPU proof-of-concept qualifies an actual two-GPU deployment.
| Strategy, devices | Modelled weights per device | Step with the gloo link | through the host | scale-up (assumed latency) |
|---|---|---|---|---|
| tensor, 2 | 7.48 GiB | 33.2 ms | 28.3 ms | 26.9 ms |
| tensor, 4 | 3.74 GiB | 34.5 ms | 20.7 ms | 16.7 ms |
| tensor, 8 | 1.87 GiB | 56.1 ms | 24.7 ms | 15.4 ms |
| pipeline, 2 | 7.48 GiB | 51.4 ms | 51.4 ms | 51.3 ms |
| pipeline, 4 | 4.23 GiB | 51.5 ms | 51.4 ms | 51.3 ms |
| pipeline, 8 | 2.60 GiB | 51.8 ms | 51.5 ms | 51.4 ms |
(The modeled 51.3 ms is an idealised batch-one, full-model memory-traffic estimate on a hypothetical sufficiently large 303 GB/s device; it is not an observed execution time. The pipeline token path reads the model’s layers in series. The table’s per-stage weight figures are approximate planner maxima: its plan() uses ceil(layers / stages) and charges embedding/head costs to a stage without constructing the exact stage-specific resident allocation.)
Pipeline partitioning can make an over-large model runnable across individually smaller devices, but for one stream its idealised latency still includes all stages plus link crossings; it is not guaranteed to equal the large-device time. its stages’ extra throughput appears only with several sequences in flight. In these hypothetical estimates, tensor parallelism may reduce batch-one token latency until link overhead dominates: with the slow gloo-class link, 8 devices are slower than 2 (56 against 33 ms), because the modeled 64 layer reductions plus final gather outweigh the modeled weight-streaming savings; with the fast link, 8 devices are faster than 4 (15.4 against 16.7 ms), barely.
How much latency per collective can a split afford? Tensor parallelism over \(p\) devices beats one device that could hold the model approximately if its average all-reduce time is below \((T_1-T_{p,\text{no-comm}})/(2L)\), where \(T_{p,\text{no-comm}}\) is the modeled per-rank time without communication. Replacing that difference by \(T_1(1-1/p)\) additionally assumes perfectly proportional division of streamed data. Both forms ignore the final logit gather, launch overhead and incomplete head/embedding sharding, so the table is an optimistic budget rather than a deployment threshold. For two devices:
| Model | All-reduces per token | Largest time per all-reduce for a 2-device split to win |
|---|---|---|
| GPT-2 | 24 | 27 µs |
| Llama-3.2-1B | 32 | 134 µs |
| Qwen3-1.7B | 56 | 115 µs |
| Mistral-7B | 64 | 381 µs |
| Llama-3.1-8B | 64 | 401 µs |
The larger the model, the more a collective can cost before the split stops paying, because the compute it hides is larger while the number of collectives stays at two per layer. The measured two-process CPU gloo all-reduce (132–140 µs) exceeds GPT-2’s modeled 27 µs budget. It is below the modeled 401 µs per-reduction budget for Llama-3.1-8B, but that numerical comparison does not establish a faster multi-GPU service: gloo loopback is not the proposed GPU collective, the final gather and residency are omitted from the break-even bound, and the assumed independent GPU bandwidth was not verified. Whether any real interconnect clears these thresholds is a question for hardware I did not measure.
The number of KV heads limits the split. Tensor parallelism divides the KV cache by splitting its heads. Llama-3.1-8B, Mistral-7B, Llama-3.2-1B and Qwen3-1.7B have 8 KV heads. Under simple, non-sequence-sharded GQA head partitioning, at most eight groups can hold disjoint KV heads; beyond that degree, KV-head replication is needed and KV bytes per rank no longer fall in proportion to tensor-parallel rank count. Multi-query attention has no KV-head dimension to split, although sequence/context parallelism could distribute its state in a different way. Thus KV-head count constrains this head-parallel design, not parallelism in general. The cost model approximates replication with a fraction rule and does not validate every candidate tensor-partition degree on real hardware.
What the numbers do not show
- No GPU was split. The distributed runs are CPU processes sharing a memory system. They show matching outputs on the tested GPT-2 greedy cases and the measured latency of the local gloo operations, and that decode on a shared memory system does not speed up. They do not show how much a GPU box gains.
- Loopback TCP is not an interconnect. Its latency (tens of microseconds) and bandwidth (about 1.2 GB/s in the larger point-to-point messages) belong to this operating system’s network stack and to gloo. NCCL, NVLink and InfiniBand were not measured.
- The collective model is validated only for small messages and few processes. It predicted the 4-process latency from the 2-process one, missed the 8-process one by a factor of two, and failed for messages over a megabyte.
- The modelled table is conditional arithmetic, not a hardware sizing certificate. It uses a roofline estimate of decode, three link parameters, one of which has an assumed latency, and ignores overlap of communication with compute, batching, runtime scratch space, allocator costs, per-stage memory asymmetry, and the sampling step.
- One model, float32, greedy, with full-checkpoint worker loading. Exactness was checked for GPT-2 in float32 with greedy decoding.
- Noise. Three runs per cell on a desktop with other work; the tensor-parallel step times have ranges that overlap between neighbouring sizes.
- Expert parallelism’s compute was not built. Only a synthetic all-to-all was measured.
- Pipeline estimation has an uncorrected helper.
pipeline_round_time()usesmax(sequences, stages), not the usualsequences + stages − 1fill/drain intervals; derived multi-sequence stage-throughput forecasts must be recalculated. - Memory placement is not proven. Full checkpoint loads occur in every worker before partitioning, and the planner’s
weights_per_deviceomits or approximates some embeddings, stage allocation and runtime overhead.
Inference Lab increment
examples/ch28/parallel_lab.py adds:
Link, and the collective-time functionsp2p_time,allreduce_time,allgather_time,alltoall_time, withallreduce_wire_bytesandfit_line.ModelShape,shape_from_configandplan: parameter counts that match real checkpoints, modelled stored against streamed bytes and KV sharing by head (not measured allocated GPU footprints), and the cost of a decode step under data, tensor, pipeline and expert strategies;tp_break_even_allreduce_s;pipeline_round_time(currently an optimistic proxy, not a validated fill-and-drain schedule).GPT2Part: GPT-2 by any share of the heads and the width, and any range of layers, withComm, a communicator that counts.run_task: a launcher of gloo process groups, and the tasks that measure CPU collectives, tensor-parallel and pipeline-parallel generation, replicas and CPU memory streaming. Every partitioned worker loads the full reference checkpoint before building its part.- Commands
shapes,plans,collectives,collectives_fine,membw,tp,pp,dp,ep,linkandgates.
To check the machine and run the split model:
python examples/ch28/parallel_lab.py gates
python examples/ch28/parallel_lab.py tp
The rule this adds: a parallel layout is part of the executable deployment manifest because it affects memory placement, communication order and numerical results—even when selected greedy outputs match, and a split is justified by what each device brings, not by how many devices there are. The cost model says what to expect, the parity check states which selected outputs the split preserved, and the measurement says whether the machine provides what the model assumed.
What you should now be able to do
- Say what each of the four strategies splits and, for a given model, how many collectives of what size one decoded token needs.
- Write the ring all-reduce and all-gather times in terms of a latency and a bandwidth, and say for which message sizes the latency alone decides.
- Check a parameter-count formula against a checkpoint’s header, and explain a mismatch.
- Test whether a particular split preserves reference outputs and logits within a specified tolerance, and explain why matching tokens neither proves identity for all inputs nor implies speed.
- Decide, for a model and a device, whether sharding is needed for weight capacity, estimate the collective latency budget, and identify the unmeasured runtime-memory and topology requirements before claiming it fits.
Key terms
- Data parallelism (in inference): replicas of the whole model serving different requests; no gradients, no collectives.
- Tensor parallelism: each layer split across devices, with collectives inside it.
- Pipeline parallelism: layers split into stages with activations passed between them.
- Expert parallelism: experts of a mixture layer placed on different devices, with an all-to-all to move tokens.
- Collective: a communication in which all devices take part (all-reduce, all-gather, all-to-all).
- Alpha–beta model: an idealised message time equal to startup latency plus bytes divided by effective bandwidth, valid only in the range and transport where its constants apply.
- Stored versus streamed bytes: the modeled resident weight and KV footprint versus the modeled bytes visited by one step; neither is a direct observation of allocator-reserved memory or DRAM transactions.
- Scale-up domain: the group of devices joined by the fastest interconnect.
Reader exercises
- Change
task_tpandtask_ppto load only their actual shards instead of first loading the full checkpoint. Measure peak and steady-state RSS per rank and verify logits and token parity again; then compare memory channels or nodes. - Add context parallelism or sequence parallelism for prefill and measure what changes in the collectives.
- Replace gloo with NCCL on a multi-GPU machine, rerun
collectives, and fit the two constants. - Overlap the all-reduce of one layer with the compute of the next micro-batch and measure how much of the communication is hidden.
- Build the expert-parallel layer from Mixture-of-Experts Inference and Expert Routing and Load Balancing on top of the all-to-all and check it for exactness.
- Fix
pipeline_round_timeto modelm + p − 1fill/drain intervals for independent token waves, add stage timelines and tests, and compare the revised throughput predictions with the measured pipeline executions. - Find the cause of the 12 to 14 ms plateaus in the 2-process collectives.
- Add explicit stage-by-stage weight, embedding, KV and workspace accounting to
plan. Compare its estimates against real device allocations, then write manifest fields recording rank topology, weight sharding and communication backend.
References
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism — Shoeybi et al.; the column and row partitions, two all-reduces per layer in the forward pass, vocabulary-split embeddings; a training paper, and its throughput figures are the authors’; read on 22 October 2026.
- GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism — Huang et al., NeurIPS 2019; micro-batching, fill/drain and the bubble overhead of a training pipeline. The forward-stage timing is relevant, but its training throughput cannot be adopted as a decode measurement; read on 22 October 2026.
- Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM — Narayanan et al., SC 2021; composition of tensor, pipeline and data parallelism, and scheduling-related throughput trade-offs. Its figures concern training and have not been reproduced in this inference lab.
- NCCL: Collective Operations — the collectives and their semantics; the page does not describe algorithms or bandwidth measures; read on 22 October 2026.
- PyTorch distributed communication package (2.6) — the backend support table; first half of the page read on 22 October 2026.
- DeepInfer at commit 774ce244 — first-party README on supported parallelism and benchmark hardware, as the authors state it; not run.
Evidence in this chapter
| Claim | Category |
|---|---|
| Tensor parallelism over 2 to 6 processes and pipeline parallelism over 1 to 4 stages reproduce the selected GPT-2 greedy tokens; the largest prompt-logit difference is 9.2e-5 | Book result (float32, CPU, 3 runs per cell; workers load full checkpoint before sharding) |
| A tensor-parallel decode step makes 24 all-reduces and one all-gather | Book result; Megatron’s two per layer is Established background |
| A small gloo all-reduce costs 132 to 140 µs at 2 processes, 339 to 362 µs at 4, 1.59 to 1.68 ms at 8; flat in size up to 64 KB | Book result (loopback TCP, this machine) |
| The two-constant model predicts the 4-process latency from the 2-process one, misses the 8-process one by a factor of two, and fails above 1 MB | Book result |
| Tensor-parallel decode of GPT-2 took 23 ms on one process and 54 to 59 ms on six; the non-communication time stayed at 21 to 29 ms | Book result |
| Eight independent replicas gave about the aggregate throughput of one (38 to 45 against 40 tokens/s) | Book result |
| Memory streaming across 2 to 12 processes plateaued at about 25 GB/s | Book result |
| A pipeline of 2 or 4 stages shows no repeatable single-sequence improvement and modest aggregate throughput gains with more sequences | Book result (CPU shared resources; separate from the uncorrected pipeline_round_time model) |
| The parameter formula matches seven checkpoints exactly | Book result |
| Host-device link: about 10 µs latency and 12.4 GB/s | Book result (this machine) |
| Llama-3.1-8B fp16 weights exceed one 12 GiB card; the modeled partition footprints and 15–56 ms/token estimates omit real runtime overhead | Modelled only (roofline, assumed independent GPU bandwidth and link parameters; no deployment fit test) |
| The largest time per all-reduce at which a 2-way split pays is 27 µs for GPT-2 and 401 µs for Llama-3.1-8B | Modelled |
| The Megatron and GPipe partitions and bubble formula | Reported by the papers; training setting; not reproduced |
| DeepInfer’s supported parallelism and benchmark hardware | Reported by the authors’ README; not run |
| NCCL, multi-GPU, multi-node, NVLink, GPU tensor or pipeline parallelism; shard-only process loading and allocated-memory fit tests | NOT_RUN |
Implementation status
- Manuscript:
content/books/inference/28-chapter.md - Runnable examples:
examples/ch28/topology.py(original toy accounting, unchanged),examples/ch28/parallel_lab.py(cost model, shapes, split GPT-2, launcher, experiments) - Tests:
tests/test_ch28_topology.py(toy),tests/test_ch28_parallel_lab.py(33 tests: the collective formulas and the fit, the starter’s figure against a ring all-reduce, the pipeline round-time helper’s currently encodedmax(m, p)rule (which needs correction for full fill/drain timing) and the GPipe bubble formula, the parameter formula against tiny GPT-2 and Llama models including tied, untied and grouped-query layouts, Gemma and Qwen3 conventions, the checkpoint counter’s handling of buffers and duplicate copies, the KV sharing rule, each strategy’s analytical plan including stored against streamed bytes, the break-even equation, an invalid split refused, the unsplit part against the library (not every partitioning and topology), and, when gloo and four CPU threads are available, real CPU multi-process runs of the collectives, tensor parallelism over four ranks and pipeline parallelism over two stages for exactness; no GPU needed) - Evidence:
research/ch28/sources.md,research/ch28/results.json,research/evidence-register.md - Verification command:
python -m unittest discover -s tests -v(setCH28_SKIP_DISTRIBUTED=1to skip the multi-process tests)
The chapter is a draft with draft = false. It makes no empirical claim about multi-GPU or multi-node speed, NCCL, or independently provisioned device bandwidth. Before promoting its projections to deployment guidance, the pipeline timing helper must be corrected and the planner’s memory figures verified against genuinely sharded checkpoint loading and full runtime allocations. Prefill/Decode Disaggregation asks what it costs to move the state of a request between machines, which is where this chapter’s link arithmetic is used.