How Continuous Batching Affects KV Cache Capacity in LLM Serving

Continuous batching—also known as iteration-level or in-flight batching—fundamentally transformed Large Language Model (LLM) serving by eliminating the coarse request-level boundaries of static batching. In traditional serving architectures, processing was constrained by the slowest sequence in a batch, leaving valuable Tensor Cores underutilized while waiting for token generation to complete. Continuous batching dynamically admits new prompt requests at every token decode step and evicts completed sequences instantly, dramatically increasing overall throughput. However, by removing the artificial compute scheduling barrier, continuous batching immediately collides with a far more unyielding physical bottleneck: GPU High Bandwidth Memory (HBM) allocated to Key-Value (KV) cache tensors.

The Capacity Accounting Behind Continuous Batching in LLM Inference

Continuous batching eliminates static sequence padding by scheduling iteration-level decoding, which shifts the primary throughput bottleneck to available GPU High Bandwidth Memory (HBM). Because each admitted request reserves an expanding KV cache block pool proportional to its active context length, serving concurrency is fundamentally governed by available memory divided by per-sequence KV footprint rather than compute execution saturation.

Under continuous batching, memory cannot be allocated statically. While model weights remain fixed in GPU VRAM, each admitted sequence consumes a dynamic volume of memory that grows monotonically with every generated token. In autoregressive Transformer architectures, self-attention mechanisms require storing the Key and Value projection states of every prior token to compute attention for subsequent tokens without redundant recomputation. The exact memory footprint consumed per token is governed by model architecture parameters:

// Exact mathematical formula for KV cache memory footprint per token
Memory per Token (Bytes) = 2 * n_layers * n_heads_kv * d_head * b

In this formulation, the factor of 2 accounts for separate Key and Value tensor allocations, n_layers represents the total number of Transformer decoder layers, n_heads_kv denotes the number of key-value attention heads (which Grouped-Query Attention, or GQA, significantly reduces compared to query heads), d_head is the head dimension (typically 128), and b is the numeric precision byte size (2 bytes for FP16 or BF16, and 1 byte for FP8). For example, consider Llama 3 70B operating with Grouped-Query Attention across 80 layers, 8 KV heads, and a head dimension of 128. In 16-bit precision, each token requires exactly 327,680 bytes (approximately 0.32 MB). At an 8,192-token context window, a single concurrent user sequence consumes over 2.62 GB of dedicated GPU memory purely for its KV cache.

On an enterprise compute node equipped with 8x NVIDIA H100 80GB accelerators (providing 640 GB of total HBM), running Llama 3 70B under Tensor Parallelism (TP=8) requires approximately 140 GB simply to load model weights. After reserving working space for CUDA kernels, activation tensors, and communication buffers, approximately 430 GB remains for the dynamic KV cache block pool. At an average context length of 8k tokens, the physical hardware is mathematically incapable of sustaining more than 164 concurrent active sequences simultaneously—regardless of how much compute headroom remains idle on the Streaming Multiprocessors (SMs).

Engine Configuration, Allocation Stalls, and Production Failure Modes

When the allocated KV block pool exceeds threshold watermarks under traffic bursts, inference engines experience allocation stalls that trigger either sequence preemption via recomputation or severe Time-to-First-Token (TTFT) queue spikes. Operators must tune engine runtime parameters including gpu_memory_utilization and max_num_seqs to balance memory allocation headroom against tail latency stability.

Modern inference runtimes like vLLM manage this dynamic memory pressure through PagedAttention, which allocates KV cache memory in fixed-size blocks (typically 16 or 32 tokens) analogous to virtual memory paging in operating systems. While PagedAttention successfully eliminates external memory fragmentation, it does not create additional physical memory. When sudden prompt bursts cause aggregate token demand to breach available block pool capacity, serving engines enter severe degraded states. Platform operators must configure engine admission parameters precisely to prevent catastrophic failure loops:

# Production vLLM serving configuration with optimized KV cache memory governance
python3 -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-70B-Instruct \
    --tensor-parallel-size 8 \
    --gpu-memory-utilization 0.90 \
    --max-num-seqs 256 \
    --block-size 16 \
    --max-model-len 8192 \
    --swap-space 32 \
    --disable-log-requests

The parameter --gpu-memory-utilization dictates what fraction of total GPU memory the engine reserves for model execution and KV caching. Setting this value to 0.90 ensures that 10% of VRAM remains unallocated to absorb runtime activation spikes and CUDA graph memory allocations without triggering unrecoverable Out-Of-Memory (OOM) kernel panics. The --block-size parameter determines paging granularity; smaller block sizes (16 tokens) minimize internal memory fragmentation for short requests, whereas larger blocks (32 tokens) reduce kernel launch overhead during multi-head attention computation.

Architectural Mitigations: KV Quantization, Prefix Caching, and Tiered Offload

Overcoming the physical HBM capacity wall requires combining KV cache FP8 quantization with architectural offloading to high-speed NVMe-oF parallel storage layers. By leveraging GPUDirect Storage zero-copy memory paths, inactive sequence states can be evicted to persistent NVMe fabrics and restored without host CPU memory bus contention, expanding practical concurrency beyond standalone GPU memory boundaries.

To scale concurrent serving capacity without purchasing exponentially more GPU nodes, production ML platforms employ three complementary architectural optimizations:

1. KV Cache FP8 Quantization: Compressing Key and Value projections from 16-bit floating-point (FP16) down to 8-bit formats (E4M3 or E5M2) cuts per-token memory consumption by exactly 50%. On Llama 3 70B, this reduces per-token consumption from 0.32 MB to 0.16 MB, instantly doubling the sustainable concurrent token capacity on an 8x H100 cluster with minimal impact on model evaluation perplexity.
2. Automatic Prefix Caching (APC): For multi-turn conversational agents, complex system prompts, and few-shot retrieval tasks, distinct user queries frequently share identical prefix tokens. PagedAttention engines hash prompt token blocks to share read-only KV cache allocations across hundreds of concurrent requests, freeing massive memory buffers for independent decode generation.
3. Tiered Offload via High-Throughput NVMe-oF: When traffic spikes inevitably exhaust even quantized HBM pools, engines must evict inactive or low-priority sequence states. Traditional offloading mechanisms swap KV blocks to host CPU DDR memory over PCIe links, causing severe host memory bus contention and CPU bounce-buffer stalls that degrade generation latency by 5x to 10x.

Enterprise platforms like OneSource Cloud solve this memory tiering dilemma at the physical infrastructure layer. OneSource Cloud provides dedicated bare-metal GPU clusters integrated with ultra-low latency NVMe-oF parallel storage and GPUDirect Storage (GDS) zero-copy data pipelines. Under this architecture, overflow KV cache blocks bypass host CPU memory bottlenecks entirely, streaming directly between GPU VRAM and high-speed NVMe flash arrays. This infrastructure design extends sustainable serving concurrency boundaries while maintaining deterministic, SLA-compliant latency profiles.

Frequently Asked Questions

How do I calculate the required KV cache pool budget for a target concurrency level?

Multiply the target concurrent sequence count by the per-token KV memory footprint of your model architecture, scale by your 95th-percentile sequence context length, and add a 20 percent safety buffer to prevent preemption during traffic spikes.

How does OneSource Cloud storage support KV cache offloading when the pool runs out?

OneSource Cloud pairs dedicated GPU nodes with tiered NVMe-oF storage and GPUDirect Storage zero-copy paths, so KV overflow moves directly to high-throughput NVMe fabric without CPU bounce-buffer stalls, expanding concurrency far beyond standalone HBM limits.