Designing the Inference Model Weight Cache for LLM Serving

Every serving incident with a memory signature traces to the same two consumers fighting over the same gigabytes: model weights, fixed and knowable, and the KV cache, growing with every concurrent context. The research options are well documented — offload to CPU, quantize the cache, spill to storage — but the design procedure that sequences them for a specific deployment is scattered across papers. This page assembles it: budget, decide, place, validate.

Prerequisites: Know Your Two Consumers

Two consumers compete for every serving GPU's memory: the model weights, fixed per deployment and known at load time, and the KV cache, which grows with concurrency and context — on the order of one megabyte per token for MHA-class models, meaning a single long-context session can consume gigabytes before the batch fills. Before designing anything, compute both numbers for your model: weight footprint by precision, and KV growth rate by architecture, because every later decision — batch limits, quantization, offload — is arithmetic on these two inputs, and serving incidents are usually budget arithmetic nobody did.

The two numbers interact exactly once in the budget — subtract both from capacity — and everywhere else in the design: quantization shrinks one or both, offload moves KV out of the equation, and batch caps ration the product. Teams that skip the arithmetic end up rediscovering it in production dashboards, which is a more expensive classroom.

The Design Steps: Budget, Decide, Place

Design in three steps. Budget: subtract weight footprint and runtime overhead from GPU memory; the remainder is the KV envelope, which sets the concurrency-context product you can serve. Decide: under pressure, apply options in order — KV quantization first (FP8 to NVFP4-class cuts KV footprint roughly in half against FP8 baselines per published work), then weight quantization if the envelope still binds, then offload tiers (CPU memory for warm sessions, storage offload with RDMA-style acceleration for multi-turn scale). Place: pin the hot tier to GPU memory, warm tiers to CPU, and the cold and checkpoint tiers to parallel storage — NVMe-oF-class tiers with GPUDirect paths of the kind delivered fabrics provide — so cache refill does not become the stall the design was meant to prevent.

The decision order encodes a principle worth stating: touch the growing consumer first. KV cache quantization halves the thing that scales with traffic without uniformly changing model outputs; weight quantization changes outputs everywhere and belongs later, behind accuracy evaluation on your own workloads. Offload tiers come last because they trade latency for capacity — a trade you should only make after cheaper arithmetic has been exhausted.

Verification: Validate Under Production Patterns

Validate with replayed production patterns, not synthetic averages: drive the serving stack with your traffic's context-length distribution and concurrency profile, and watch three indicators — KV admission rejections at peak (envelope too small), cache-refill latency after eviction (placement wrong), and tail latency drift across a multi-turn session chain (offload tier too slow). Fix by tier before touching the budget: refill problems are storage-path problems, rejection problems are envelope problems, and conflating them produces quantized models nobody needed. The validated artifact is a configuration plus a load-report showing the three indicators within targets at P95 traffic.

The storage path is where designs quietly die: an offload tier on slow storage converts a memory win into a latency loss exactly when multi-turn chains need their history. The delivered answer to that path is parallel NVMe-oF tiers with GPUDirect Storage zero-copy paths — the storage architecture behind OneSource Cloud AI Storage Architecture — which keeps warm, cold, and checkpoint tiers refilling fast enough that offload remains a win.

FAQ

Should you quantize the KV cache or the model weights first?

Under serving pressure, the KV cache first: KV quantization (FP8 and below) roughly halves the growing consumer without touching model quality uniformly across uses, while weight quantization changes answers everywhere — quantize weights only when the envelope still binds after KV measures, and with accuracy evaluation for your workloads, not vendor benchmarks.

When is storage offload for KV cache worth the complexity?

For long-context and multi-turn patterns: when session chains keep caches warm across turns and re-prefill costs dominate, storage offload with RDMA-class acceleration pays for itself — and on dedicated enclaves the NVMe-oF tiers with GPUDirect paths that OneSource Cloud delivers give the warm and cold tiers a refill path that keeps offload from becoming the new stall.