How Continuous Batching Works in LLM Inference Serving

Continuous batching — also called iteration-level batching — admits and evicts LLM inference requests mid-generation rather than waiting for an entire batch to finish, keeping the GPU saturated as variable-length requests complete and new ones arrive. It is the mechanism that makes batching work for the uneven, conversational traffic typical of LLM serving, and it is the modern standard for efficient serving. For the broader batching framework, see LLM inference batching.

For teams deploying LLM serving, understanding how continuous batching works is essential because it is the single largest throughput lever in the serving stack. A serving system that implements continuous batching extracts multiples of the throughput from the same GPU as one that does not, and knowing the mechanism — and its tradeoffs — is what lets you configure it for your workload rather than accepting defaults. This guide explains the algorithm, why it beats static batching, and the latency-throughput tradeoffs it entails.

Static Batching and Why It Fails for LLMs

Static batching groups a fixed number of requests into a batch, processes the entire batch together, and waits for every request in the batch to complete before starting a new one. This works well when all requests are roughly the same length — the batch completes together, and the next batch starts immediately. For LLM serving, where requests produce variable-length outputs (a short answer finishes quickly; a long generation runs for hundreds of tokens), static batching breaks down: the batch waits for its longest request to finish, leaving finished slots idle, while new requests queue outside waiting for the whole batch to complete.

The result is low GPU utilization despite high queued demand — the GPU is busy only for the overlapping portion of the batch, and finished slots waste capacity waiting for the straggler. For the variable-length traffic typical of conversational AI, static batching leaves substantial throughput on the table, which is why continuous batching was developed.

How Continuous Batching Works

Continuous batching operates at the iteration level — one forward pass of the model — rather than at the request level. At each iteration, the serving system assembles a batch from the currently active requests: those that have been admitted and have not yet finished generating. When a request finishes (generates its end-of-sequence token), it is evicted from the batch immediately, and a waiting request can be admitted to take its place at the very next iteration. The batch size stays full as requests come and go, hence "continuous."

This dynamic admission and eviction is what makes continuous batching fit variable-length traffic. Short requests finish quickly and vacate their slots; long requests stay in the batch as long as they need; new requests fill vacated slots without waiting for the whole batch to drain. The GPU stays saturated with active work across the batch, extracting far more throughput than static batching from the same hardware. For the underlying KV cache mechanics that enable this, see what is the KV cache.

Continuous vs static batching comparison

The Throughput-Latency Tradeoff in Continuous Batching

Continuous batching raises throughput substantially, but it introduces latency dynamics that must be managed. When the batch is full — meaning all slots are occupied and new requests must wait — the waiting requests experience queue latency that grows with queue depth. When the batch is below capacity, waiting requests are admitted immediately. The tail latency depends on how often the batch is full and how long the queue is when it fills.

The maximum batch size is the tuning parameter: a larger max batch size extracts more throughput but fills the batch more often, increasing the chance a new request waits; a smaller max batch size limits throughput but reduces queue latency. The right setting balances throughput against the workload's latency target, and it should be tuned under real traffic, not just to the GPU's memory limit. For how to diagnose when batch size is the latency cause, see what causes high p95 latency.

Memory Management Under Continuous Batching

Continuous batching depends on efficient memory management, because admitting and evicting requests at every iteration means the KV cache — which holds attention state for each active request — is constantly being allocated and freed. Paged attention, where the KV cache is allocated in small pages rather than one large block per request, is the technique that makes this efficient: when a request finishes, its pages are returned to a pool immediately; when a new request starts, it draws from the pool. Without paged attention, continuous batching would waste memory on the idle slots of evicted requests.

The practical implication is that continuous batching and paged attention are complementary — paged attention provides the memory efficiency that makes continuous batching's frequent slot recycling practical, and together they are the standard modern serving architecture. For how the KV cache drives memory and latency, see our KV cache explainer.

FAQ

How does continuous batching work in LLM serving?

At each iteration (one forward pass), the server assembles a batch from currently active requests. When a request finishes generating, it is evicted immediately; a waiting request can be admitted at the next iteration to fill the vacated slot. The batch stays full as requests come and go, unlike static batching where the whole batch must complete before any new request starts. This keeps the GPU saturated across variable-length requests, extracting far more throughput.

What is the difference between continuous and static batching?

Static batching processes a fixed batch until every request completes, then starts a new batch. Finished slots go idle waiting for the slowest request. Continuous batching admits and evicts requests at every iteration, so slots recycle as soon as a request finishes. For variable-length LLM traffic, continuous batching achieves much higher GPU utilization and throughput. See the comparison table above.

Does continuous batching increase latency?

It introduces the possibility of queue latency when the batch is full — new requests must wait for a slot to open. The maximum batch size determines how often this happens: a larger max batch raises throughput but fills the batch more often. Tune the max batch size to the workload's latency target to balance throughput and tail latency. For diagnosis of latency causes, see what causes high p95 latency.

How does paged attention enable continuous batching?

Paged attention allocates the KV cache in small pages rather than one large block per request, so when a request finishes, its pages return to a pool immediately. Without it, continuous batching's frequent slot recycling would waste memory on idle slots. Together, continuous batching handles the scheduling and paged attention handles the memory — they are the complementary techniques of modern LLM serving. For the KV cache mechanics, see our KV cache guide.

Summary

Continuous batching admits and evicts LLM inference requests at each iteration, keeping the GPU saturated as variable-length requests finish and new ones arrive. It beats static batching by recycling slots immediately rather than waiting for the whole batch to drain, which dramatically raises throughput for conversational traffic. It introduces queue latency when the batch is full, which the maximum batch size parameter controls — tune it to the workload's latency target. Paged attention provides the memory efficiency that makes continuous batching's frequent recycling practical. Together they are the modern serving architecture. For the full framework, see LLM inference batching.