Continuous Batching

Under a light load, three tenths of the capacity of the server, requests to a statically batched GPT-2 waited a median of one to one and a half seconds for their first token. The same requests to the same model on the same GPU with continuous batching waited 55 to 60 milliseconds. At a load near capacity the median wait for a first token was five to six seconds for the static server with a batch of 8, and 1.4 to 2.4 seconds with a batch of 32. The continuous server’s was still 60 to 70 milliseconds.

Against a stated objective, a first token inside one second and no more than 100 milliseconds per token after it, the continuous server met the objective for every request in every run I made. The static server met it for between 6 and 52 per cent of them.

Nothing about the model differs between the two. The weights, the cache, the decode function and the arithmetic are the same, and so, as the tests check, is every generated token. The only difference is when the scheduler is allowed to change who is in the batch. This chapter measures what that one decision is worth, what it costs, and what it does to fairness, with the model running on the GPU and the clock counting the real time of every iteration.

I also have a confession about the harness that produced these numbers. It had a bug that made two experiment runs hang, one for 38 minutes and one for about two hours, with no output at all. The bug was a rounding error in the scheduler’s clock, and it is described near the end because it is a good example of a failure that no test I had written would have found.

Where this sits

Chapter 9 split a request into prefill and decode, defined the latency metrics, and showed in a toy alternation of two streams that one request’s prefill stalls another’s decode. Chapter 11 found that decoding up to about 32 sequences together costs only 7 to 20 per cent more per step than decoding one. Chapter 14 built the paged cache that lets a set of variable-length sequences share memory. This chapter assembles them into a server: requests arrive over time, and a scheduler decides at each step who runs.

The Inference Lab piece added here is the serving loop, examples/ch15/serving_loop.py: a seeded Poisson request generator; a StaticScheduler and a ContinuousScheduler over one paged cache and one decode function; two admission policies; chunked prefill; and the metric set of Chapter 9 applied to every request. It runs real GPU time with a virtual clock for idle gaps, so an experiment lasts seconds and its latencies are the ones a client of this machine would see.

For the Pi series, this is the chapter that turns an engine into something with a service level. An agent that waits seconds for the first token of each step is a slow agent, whatever the model’s quality, and the next chapters on APIs and protocols (Part V) start from the figures here.

Two ways to share a batch

Static batching. The server collects up to \(B\) requests (or waits until the oldest has waited long enough), prefills them, and decodes them together until the longest has finished. Only then does it form the next batch. A request that finishes early still occupies its slot, and the decode step still computes a token for it, which is discarded. A request that arrives while a batch is running waits for all of it.

Continuous batching (iteration-level scheduling). At each iteration the scheduler looks at the state of the world and decides: admit waiting requests while there is room in the batch and memory in the cache; prefill the ones just admitted; decode one token for every request that is already generating; remove any that have finished. The batch changes every iteration. A finished request leaves immediately and a newcomer can take its place in the next iteration.

Orca introduced this under the name iteration-level scheduling. Its abstract describes a scheduler that runs “one iteration” of the model at a time on the batch, lets early-finishing requests return, and lets newly arrived requests avoid waiting for the whole current batch. The same abstract reports a 36.9-fold throughput advantage over FasterTransformer on a 175-billion-parameter model at the same latency, a figure about a system and model far from this one, which I have not tried to reproduce.

In ContinuousScheduler.run, each iteration does four things in order: admit (reserving each request’s blocks so that it cannot run out of cache mid-sequence), prefill (the whole prompt, or a bounded number of tokens if chunking), a ragged decode step for every active request using the Chapter 14 function, and bookkeeping of timestamps and finished requests. Every token produced in an iteration is stamped with the time at the end of that iteration.

What is being compared

Both schedulers use the same paged cache, the same decode function and the same per-request prompts. The tests check the property that matters: for a mix of requests arriving at different times with different lengths, each request’s generated tokens are exactly those it would produce alone, for both schedulers and also with chunked prefill. Batching changes when work happens, never what is computed.

The experiment

Workload. Sixty-four requests per run, arriving as a Poisson process. Prompt lengths are log-normal with a median of 60 tokens, capped at 300; output lengths log-normal with a median of 40, mean 51, capped at 128. Prompts are random token ids, since only lengths matter for timing.

Offered load. I first measured the saturated throughput of the continuous scheduler by presenting 96 requests at once:

I took 543 tokens per second, or 10.6 requests per second at the workload’s mean of 51 output tokens, as capacity, and drove the server at 0.3, 0.6, 0.9 and 1.2 times that rate.

Schedulers. Static with batch sizes 8 and 32 (and a 100 ms maximum wait to fill a batch), and continuous with a maximum batch of 32.

Objective. A time to first token of at most 1 second and a time per output token of at most 100 milliseconds. I fixed these before the runs, in the code, and did not adjust them afterwards.

Repeats. Three seeds per cell. The ranges below are across those seeds.

What the clock is. Idle time between requests is skipped, so a run takes seconds. Every iteration’s duration is measured with a synchronised timer. Latencies include real compute and exclude real waiting for the network, of which there is none.

Results

The first-token delay is where the difference lives. The static server cannot start a request until it has formed a batch, and cannot form the next batch until the last one’s longest request ends. At low load that means waiting for a batch to fill (up to the 100 ms timeout) and then, for a large part of the time, queueing behind a batch that is running its longest member. As load rises, the static server’s queue grows and its delay reaches seconds. The continuous server’s median first-token delay stays at one prefill plus at most an iteration: the new request is admitted at the next iteration boundary, a few tens of milliseconds away.

The median end-to-end time is 1 to 7 seconds. It is dominated by the 40-odd decode steps of a typical request, which are the same under both schedulers; the continuous server’s advantage there is smaller, a factor of 2 to 6, and comes from not queueing.

Throughput, and a caution

Output tokens per second, ranges over three seeds:

Continuous scheduling gave 1.1 to 1.4 times the throughput of the larger static batch, and 1.1 to 2.8 times that of the smaller (1.1 at the lightest load and 2.4 to 2.8 at the heaviest). But none of the runs approaches the 543 tokens per second measured at saturation. A 64-request trace is short: it includes the time to ramp up and drain, and the mean number of active sequences in the continuous runs, 3.0 to 12.7, never reached the 20 seen when everything arrives at once. The table says how the schedulers compare on identical short traces, not what either can sustain. I did not run longer traces.

Where static batching loses

Two mechanisms are visible in the data.

Wasted slots. The decode step of a static batch continues until its longest member finishes, computing a token for every member, including those that are done. I counted the fraction of decode slots that produced a token someone wanted. It was between 0.38 and 0.59 in every static run. Between 41 and 62 per cent of the decode work was spent on requests that had already finished. Continuous batching has none of this waste by construction.

Head-of-line waiting. A request that arrives just after a batch starts waits for that batch’s longest member. With output lengths spread log-normally, the longest of eight or thirty-two is much longer than a typical one.

The larger static batch was better than the smaller on latency at every load, because it forms and drains the queue faster, and it still lost to continuous scheduling by an order of magnitude on first-token delay.

What continuous batching costs

The advantage is not free. Two measures went the other way:

Time per output token is higher because the continuous batch is larger on average and each step is a little slower. The worst inter-token gap is two to three times higher, which is Chapter 9’s interference arriving in a real scheduler: a newly admitted request’s prefill runs inside an iteration, so every request already generating waits for it. The static server’s decode steps are never interrupted by a prefill, because it prefills a whole batch first and decodes afterwards. Its inter-token gaps are smooth. It makes you wait first and then streams.

Which of the two a user would prefer depends on the use. For a chat interface the first token matters more, and a 100 ms hiccup mid-stream is hard to notice. For a pipeline reading a stream at a fixed rate it may be the other way. The goodput objective above counts both, and the continuous server met it everywhere, since even its worst gaps were far below 100 ms per token on average. A stricter objective on the worst gap, say 50 ms, would change the picture.

Chunked prefill

Chapter 9 showed that splitting a long prefill into chunks bounded the stall it inflicted on another stream, at the price of delaying the long request. I tested the same idea inside the scheduler: with 10 per cent of requests having 700-token prompts, each iteration prefills at most a fixed budget of prompt tokens, shared among the waiting prefills, and decodes everyone else normally.

Short requests’ 95th percentile of the worst inter-token gap, in milliseconds, per seed; and their 95th percentile first-token time:

The result is inconclusive on the benefit and clear on the cost. The worst-gap column does not fall consistently: for chunks of 64 it is lowest for seed 1 and highest for seed 2. With only 7 to 9 long requests per run there is too little to tell. The cost is clear and monotonic. Short requests’ first-token times rose from tens of milliseconds to hundreds, and long requests’ doubled to more than quadrupled. In my implementation the budget is shared among all requests being prefilled in arrival order, so a long prompt ahead of a short one in the queue delays it.

Why was Chapter 9’s clean result not reproduced? There the stall was 56 to 73 milliseconds against one stream’s 11 millisecond step. Here a 700-token prefill costs about 35 to 45 milliseconds against a mix of many streams and iterations of 20 to 40 milliseconds, so the stall is only a couple of iterations long and is lost in the variation between runs. Sarathi-Serve’s abstract reports large capacity gains from chunked prefills with stall-free scheduling on A100-class GPUs with 7-billion-parameter and larger models, where prefills are far longer relative to a decode step. I did not test that regime and nothing here contradicts it. The honest statement is that on a small model on a modest GPU with short prompts, chunking costs more than it saves.

Fairness

Admission order is a policy. First-come-first-served (FCFS) is simple and fair by arrival. Shortest-prompt-first (SPF) admits the cheapest waiting request, which lowers the average wait and risks starving the expensive ones. To make the queue the bottleneck I capped the batch at 8 and used 25 per cent long prompts (600 tokens). Queue wait is the time from arrival until admission.

At offered load 1.2, two seeds:

SPF cut short requests’ median wait by a factor of 3 to 8 and raised the worst wait of a long request by a factor of 1.6 to 3.4. Total throughput was the same within noise (391 and 442 tokens per second under FCFS, 398 and 474 under SPF). That is the standard trade: the policy does not create capacity, it moves waiting from some requests to others.

At load 0.9 the two seeds disagreed, with SPF helping in one and hurting both classes in the other. With the queue short most of the time, which request is chosen matters little, and what the result mostly reflects is the luck of the arrival sequence. I report the disagreement as it is.

A policy is a claim about whom to make wait, and a fairness claim needs a metric per class. A single average wait would have called SPF a clear win here. Any real deployment needs the per-class tail and an aging rule that bounds how long the unlucky class can be passed over.

The bug

Twice, an experiment run produced nothing and never ended: once for 38 minutes before I stopped it, then for about two hours. The process was alive and using one processor core. No test had failed, because none had exercised the case.

I reproduced the hang on a single configuration, and set a watchdog (faulthandler.dump_traceback_later) to print where the process was when a timer expired. It was in the static scheduler’s wait logic. When the scheduler had a partly filled batch and had to decide whether to keep waiting, it computed

ready = len(waiting) >= batch_size or not pending or now - oldest >= max_wait
if not ready:
    now = min(oldest + max_wait, pending[0].arrival)

If the next arrival was far off, now was set to oldest + max_wait. The test now - oldest >= max_wait should then be true. In floating point it can be false: for oldest = 4.222109257625241, (oldest + 0.1) - oldest is 0.09999999999999964. The scheduler then recomputed the same horizon, set now to the same value, and went round again forever.

The repair compares now with the horizon directly, and the loop now raises an error if time fails to advance. Two regression tests reproduce it with the crafted arrival time and with thirty random arrival patterns that must terminate, and I confirmed that the committed code hangs on the first. The results above come from the fixed code (and the earlier successful runs were unaffected, since the defect could only produce a hang).

It matters here for a reason beyond tidiness. A simulation in which time advances in jumps is only as sound as its comparisons of times, and equality of computed floating-point times is fragile. This particular failure was loud, but a version that exited a loop early, or skipped a wait, would have produced plausible numbers.

What the numbers do not show

• A small model on a modest GPU. Launch-bound steps of 20 to 40 milliseconds. Larger models, faster hardware and longer prompts all change the balance between prefill and decode, and the chunked-prefill result especially.
• Short traces. Sixty-four requests; throughput includes ramp-up and drain and is not saturated capacity.
• Three seeds, or two. The ranges are real variation, and the chunking and fairness experiments are small.
• Synthetic arrivals and lengths. Poisson arrivals with log-normal lengths are not a production trace.
• Reservation at admission, no preemption. Chapter 14’s on-demand blocks and watermark are not used here.
• Random prompts. Timing only; no text quality is measured.
• The objective is one choice. Different thresholds would give different goodput.
• Absolute latencies depend on the host. Launch costs varied between sessions (Chapter 11).

Inference Lab increment

The Lab now has:

• make_workload, StaticScheduler, ContinuousScheduler (policies fcfs and shortest_prompt, optional prefill budget), Server, summarise, pick_next and percentile, with 13 tests including exactness against single-request decoding for both schedulers and chunked prefill, memory returned after a run, and the livelock regression.
• The capacity, loads, chunking and fairness commands, each reporting progress as it goes.

The rule this adds: a scheduler is judged by latency distributions and an objective, by class, at stated loads, with the model’s outputs checked for equality. Throughput alone would have shown continuous batching as a modest improvement. The latency distributions showed it as a different kind of server.

What you should now be able to do

• Describe static and continuous batching and say what changes between iterations.
• Compute the useful fraction of decode slots in a static batch.
• Define offered load relative to measured capacity and design a replay that uses a virtual clock for idle time.
• Report TTFT, TPOT, worst inter-token gap and goodput by scheduler and load.
• Say what continuous batching costs in smoothness, and when chunked prefill is likely to repay its cost.
• Explain how an admission policy moves waiting between classes, and what to measure to see it.
• Find a non-terminating loop in a simulation with a watchdog traceback.

Key terms

• Static batching: a fixed batch run to completion.
• Continuous batching / iteration-level scheduling: the batch is revised at every model iteration.
• Offered load: the arrival rate as a fraction of measured capacity.
• Goodput: the fraction of requests meeting stated latency objectives.
• Useful fraction: decode slots that produced a token someone wanted.
• Queue wait: time from arrival to admission.
• Starvation: a request class waiting without bound because others are always preferred.
• Virtual clock: a simulated clock that skips idle gaps and adds real compute time.

Reader exercises

1. Compute the useful fraction analytically for a static batch of 8 with output lengths drawn from your distribution. Compare it with the measured 0.5.
2. Add aging to shortest-prompt-first: a request’s priority improves with its waiting time. What aging rate bounds the long requests’ maximum wait at twice FCFS’s?
3. Rerun loads with 400 requests. How much closer do the continuous server’s throughput figures get to the 543 tokens per second capacity?
4. Replace admission-time reservation with on-demand blocks and a watermark (Chapter 14). What does it do to the maximum batch the server can sustain?
5. Make the chunk budget go first to requests that have waited longest or have the shortest remaining prompt. Does the short-request TTFT cost fall?
6. Add a 50 ms objective on the worst inter-token gap. What happens to the continuous server’s goodput?
7. Insert a deliberate rounding error into the continuous scheduler’s clock and write the test that would have caught it.

References

• Orca: A Distributed Serving System for Transformer-Based Generative Models — iteration-level scheduling (OSDI 2022). Its abstract’s 36.9× figure is for a 175-billion-parameter model and is not reproduced here.
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — chunked prefills and stall-free scheduling (OSDI 2024); its reported capacity gains were not tested.
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — placing the phases on different resources, the subject of Chapter 29.

Evidence in this chapter

Implementation status

• Manuscript: content/books/inference/15-chapter.md
• Runnable examples: examples/ch15/scheduler.py (original abstract simulator), examples/ch15/serving_loop.py (the serving loop and experiments)
• Tests: tests/test_ch15_scheduler.py, tests/test_ch15_serving_loop.py (exactness for both schedulers and chunked prefill, memory returned, timestamp consistency, admission under a small pool, policies, workload determinism, metrics and the livelock regression; all on a small model, no GPU needed)
• Evidence: research/ch15/sources.md, research/ch15/results.json, research/ch15/raw/, research/evidence-register.md
• Verification command: python -m unittest discover -s tests -v

The chapter is a draft with draft = false. It makes no claim about preemption, priorities, large models or production traffic. Chapter 16 stops treating each request as independent: requests that share a prefix can share the work of processing it, and the interesting questions become when that is correct, how long to keep it, and who may see that it happened.