Disaggregated Prefill and Decode for Enterprise LLMs: KV Cache RDMA Transfer, TTFT, and Serving Economics

NoraLin 46 2026-11-19 19:08:14 Edit

In high-throughput enterprise Large Language Model (LLM) serving architectures, achieving both low Time To First Token (TTFT) and high token generation throughput (Time Per Output Token [TPOT]) has emerged as a fundamental systems challenge. Conventional serving runtimes co-locate both phases of autoregressive generation on the same physical GPUs. However, prompt prefill and token decoding possess diametrically opposed computational and memory access characteristics: prefill is compute-bound, saturating tensor cores through massive General Matrix Multiplications (GEMM), while decoding is memory-bandwidth-bound, executing sequential General Matrix-Vector (GEMV) operations throttled by High-Bandwidth Memory (HBM) read speeds.

When batching both phases together on shared hardware, incoming multi-thousand-token prompts monopolize GPU compute resources, starving active decoding sequences and causing catastrophic p99 latency spikes. To resolve this structural contention, modern inference architectures deploy disaggregated prefill and decode frameworks (such as Mooncake, Splitwise, and DistServe). By physically separating prompt computation from token generation and migrating serialized Key-Value (KV) cache tensors across high-speed RoCEv2 networks, disaggregation delivers deterministic tail latency and unlocks hardware-level capital efficiency.

Disaggregated Prefill and Decode Architecture for Enterprise LLM Inference

Mechanics of Prefill-Decode Interference: Compute-Bound Phase vs Memory-Bound Phase

In unified LLM serving, compute-intensive prefill passes (saturating tensor core FLOPS on long prompts) interleave with memory-bandwidth-bound decode iterations, creating catastrophic prefill-decode interference that inflates Time To First Token (TTFT) and stalls token generation throughput. Understanding this interference requires analyzing the operational profile divergence across the execution lifecycle.

Computational Divergence: GEMM Saturation vs Memory Bandwidth Throttling

During the prefill phase, an input prompt of length S processes all tokens simultaneously in parallel forward passes, generating dense activations that saturate NVIDIA Tensor Core compute capacity (>80% MFU). Conversely, the decode phase generates exactly one token per iteration per sequence (S=1). At this batch dimension, execution speed is completely bounded by the rate at which model weights and KV cache tensors can be fetched from HBM3e into on-chip SRAM registers.

Prefill-Decode Resource Interference vs Disaggregated Pipeline Isolation

Architectural Matrix: Unified Serving vs Disaggregated Serving

System Dimension Unified Serving (vLLM / TensorRT-LLM Default) Disaggregated Architecture (Prefill / Decode Split)
Hardware Workload Type Heterogeneous compute-bound and memory-bound mix Homogeneous pools: Prefill (Compute-Bound), Decode (Memory-Bound)
Tensor Core Utilization Unpredictable fluctuations between 20% and 85% MFU Prefill nodes sustain >80% MFU; Decode nodes maximize HBM bandwidth
Latency Interference Severe; incoming prefill batches delay active decodes by 200-800ms Zero interference; prefill and decode execute on dedicated physical silicon
KV Cache Management Shared local VRAM pool using PagedAttention Local allocation + high-speed RDMA migration to decode pool
Scaling Mechanism Coarse-grained; adding nodes scales both phases symmetrically Fine-grained; add compute nodes for prompt burst, memory nodes for context

SLA Determinism and Capital Efficiency: Eliminating TTFT Spikes and Tail Latency

Separating prefill and decode enables strict contractual SLA isolation where long enterprise documents do not disrupt low-latency conversational generation, reducing p99 TTFT by up to 5x while allowing hardware specialization between high-compute and high-memory-bandwidth nodes. In enterprise environments where retrieval-augmented generation (RAG) and agentic workflows inject multi-thousand-token system prompts, this architectural isolation is critical.

Quantitative Advantages of Disaggregated Infrastructure

  • Contractual p99 TTFT Guarantee: Dedicated prefill pools prevent queue head-of-line blocking, stabilizing initial token delivery under 300 milliseconds even during massive traffic spikes.
  • Decode Throughput Optimization: Free from prefill interruptions, decode GPU workers maintain steady autoregressive step cadences (e.g., 25-40 tokens per second per user stream) without intermittent stalling.
  • Heterogeneous Hardware Specialization: Infrastructure architects can assign cost-effective, high-FLOPS accelerators (such as NVIDIA L40S or H100 NVL) to prefill pools, while reserving massive HBM3e capacity nodes (such as H200 141GB or B200) for the decode fleet.
  • Asymmetric Elastic Scaling: If an application experiences sudden surges in document analysis volume, operators scale only the prefill tier without over-provisioning expensive long-term memory capacity.

The inter-node network bandwidth required to transfer the populated KV cache without stalling downstream decoders is defined by the KV transfer equation:

// Required Inter-Node KV Cache RDMA Transfer Bandwidth
// BW_kv = (2 * n_layers * d_head * n_kv_heads * S_prompt * precision_bytes) / T_transfer_target
Example: 70B Model with GQA (n_layers=80, d_head=128, n_kv_heads=8, S_prompt=4096, FP16)
Total KV Cache Size per Request = 2 * 80 * 128 * 8 * 4096 * 2 bytes = 1.34 Gigabytes
Target Migration Latency: T_transfer = 10 milliseconds
Required Network Bandwidth = 1.34 GB / 0.010 s = 134 GB/s (~1.07 Tbps RoCEv2 line rate)

Operational Boundaries: Inter-Node RDMA KV Migration and the Chunked Prefill Crossover

Disaggregated serving becomes economically and architecturally superior only when clusters deploy non-blocking 400Gbps+ RoCEv2 networks and average prompt lengths exceed 2,048 tokens; below this threshold, in-node chunked prefill delivers comparable latency with zero network serialization overhead. Knowing where this economic and physical inflection point lies prevents premature over-engineering.

Inter-Node KV Cache RDMA Migration Pipeline and Memory Page Mapping

Operational Boundaries and Infrastructure Milestones

  1. Network Fabric Bandwidth Floor: Disaggregation is strictly contingent upon 400Gbps or 800Gbps RDMA networking. If inter-node bandwidth is limited to standard 100GbE, KV cache serialization and transfer latency will exceed the time saved by compute isolation, causing net latency degradation.
  2. Prompt Length Inflection Crossover: For workloads with short prompt lengths (<1,024 tokens) such as simple chatbots, chunked prefill (breaking prompts into 512-token chunks within the same engine) achieves comparable latency without network overhead. When average prompts cross 2,048 to 8,192 tokens (RAG, legal search, codebase analysis), disaggregation becomes mandatory.
  3. Zero-Copy RDMA Memory Allocators: Deployment requires unified memory management (such as Mooncake's distributed cache store or vLLM PagedAttention v2) capable of pinning remote GPU memory addresses via RoCEv2 verbs, eliminating host CPU intermediate staging.
  4. Dynamic Prefill-Decode Ratio Balancing: Platform schedulers must continuously monitor queue depths and dynamically adjust node assignments between prefill and decode roles to prevent stranded capacity under shifting diurnal traffic patterns.

OneSource Cloud deploys dedicated bare-metal GPU clusters interconnected by non-oversubscribed 400Gbps/800Gbps RoCEv2 fabrics, enabling zero-copy RDMA KV cache migration between disaggregated prefill and decode pools with microsecond-level transfer latency.

Frequently Asked Questions

What is the primary technical tradeoff between disaggregated prefill-decode and in-node chunked prefill?

In-node chunked prefill breaks long prompts into smaller token chunks (e.g., 512 tokens) and interleaves them with decode steps on the same GPU, avoiding network overhead but still causing minor compute interference. Disaggregated prefill-decode physically isolates the phases onto separate nodes, eliminating all interference and allowing hardware specialization, but requires ultra-high-speed 400Gbps+ RoCEv2 networks to transfer multi-gigabyte KV cache states without latency penalty.

How does OneSource Cloud infrastructure optimize KV cache RDMA transfers in disaggregated serving?

OneSource Cloud provides single-tenant bare-metal GPU nodes linked by dedicated non-blocking 3.2 Tbps aggregate RoCEv2 fabrics per chassis with hardware GPU-Direct RDMA, allowing multi-gigabyte KV cache states to migrate across prefill and decode workers in under 5 milliseconds with zero CPU proxying or network packet drops.

Previous: Private LLM Deployment: Infrastructure Requirements for Enterprise Teams
Related Articles