GPU Memory Planning for Long-Context LLM Inference

GPU memory planning for long-context LLM inference is sizing HBM for weights, KV cache, and concurrent sequences so a tail prompt does not OOM a replica that still looks idle on SM utilization. Prefill versus decode splits compute. Long context splits memory. If you only count GPUs, you will miss the window that actually fails.

Product teams ask for 128k context because a competitor slide has it. Serving teams discover one fat prompt evicts every other conversation on the replica. Plan the tail, the concurrency, and the refuse path. Do not invent a universal GPU count.

What actually consumes HBM

Paged attention and prefix cache reduce some waste. They do not make a 128k prompt free. Measure your prompt-length histogram. Size for a percentile you will actually accept, and publish what you will refuse. A silent OOM is worse than a 413.

Concurrency is a memory budget

One long sequence can cost as much cache as many short chats. If the SLA is many concurrent users, cap context per request or add replicas. Tensor parallel can split weights and cache across GPUs. It also needs a fabric that does not fight serving p99. Exclusive GPUs make that split a partition, not a noisy-neighbor experiment.

Do not steal training HBM for a surprise 128k demo. Reserve an inference partition. Long-context spikes look like idle SM and full memory. Reclaim that partition into training and you will page the product.

Where exclusive infrastructure helps

You can pin SKU memory size, replica count, and a refuse policy without a public quota ticket. Storage still matters if you offload KV; that path must meet residency. OneSource Cloud’s private AI infrastructure is exclusive capacity you can size for HBM, not only for SM. Keep serving on OnePlus, OneSource Cloud’s AI orchestration platform, away from training burst. AI networking matters when you shard context across nodes. Managed operations watch HBM and OOM, not only GPU util.

FAQ

Why does long-context inference OOM when GPU utilization looks low?

Because SM util is not HBM. KV cache grows with tokens and concurrency. A replica can be memory-full and compute-quiet. Plan memory from the prompt tail. Do not use a training-style utilization target as the health check for long-context serving.

How do we plan GPU memory without a fake 128k benchmark?

Use your production prompt histogram, the model precision, and the concurrent sequences you sell. Add a refuse threshold. Sample the tail on one replica. Extrapolate replicas. If you have never seen 128k in production, do not size the farm as if every request were 128k.

Does tensor parallel solve long-context memory?

It can split cache and weights across GPUs. It adds latency and fabric load. Use it when one GPU’s HBM cannot hold the tail you must serve. It is not a substitute for a concurrency cap. Measure TTFT after the split.

Should we offload KV cache to host or storage?

Only with a measured latency budget and a residency path. Host DRAM offload can save HBM and hurt p99. Disk offload is usually the wrong serving tool. If you offload, treat it as part of AI storage design, not as a hidden flag.

Can training GPUs cover a long-context spike?

Only with reclaim faster than the SLO and weights already warm. Otherwise a fine-tune owns the HBM when users arrive. Long-context serving belongs on reserved exclusive GPUs.

Summary

Plan long-context inference as an HBM problem: weights, KV cache, concurrency, and a refuse path. Exclusive partitions keep that memory off training. If you need to size that pool in a U.S. private environment, use OneSource Cloud private AI infrastructure and hold serving on OnePlus.