How Much GPU Memory LLM Inference Needs and Why It Matters

LLM inference GPU memory is consumed by three things — model weights, the KV cache, and activations — and the total determines which GPU can hold the model, how many concurrent requests it can serve, and whether the model fits at all. For the KV cache mechanics, see what is the KV cache. For how memory drives serving cost, see how to reduce LLM inference cost.

For teams deploying LLMs, GPU memory is the binding constraint more often than compute. A model that fits in GPU memory serves well; a model that exceeds it must be sharded across multiple GPUs, which adds communication overhead, latency, and cost. Understanding what consumes memory, how to estimate it, and how to manage it is the foundation for serving at scale.

What Consumes GPU Memory During Inference

Weights are the fixed cost: the model parameters loaded into GPU memory for the duration of serving. In FP16/BF16 precision, each parameter consumes two bytes, so a 7B model uses roughly 14GB for weights alone — already a significant fraction of a typical 80GB GPU. The KV cache is the variable cost that grows with context length and concurrency, and it can exceed the weight memory for long-context or high-concurrency workloads. For the cache scaling behavior, see the KV cache explainer. Activations are the per-step working memory that depends on batch size; they are smaller than weights and KV cache but add to the total. For how batching interacts with memory, see LLM inference batching.

The practical total is weights plus KV cache for the target context length and concurrency. A model that fits in 80GB at low concurrency with short context may not fit at high concurrency with long context, because the KV cache grows with both. This is why serving capacity planning must include the memory estimate at the intended context length and concurrency, not just the model in isolation. For capacity planning, see capacity planning for inference.

Estimating Per-Model Memory Requirements

A rough memory estimate for LLM inference: weight memory is approximately 2 bytes per parameter for FP16/BF16. KV cache memory per token scales with the number of layers, attention heads, and head dimension; a common estimate is in the range of several hundred KB to a few MB per token per request, varying by model architecture. Multiply that by the target context length and concurrency, and add to the weight memory. The total determines whether one GPU can hold the workload or whether tensor parallelism across GPUs is required.

For models that exceed a single GPU, multi-GPU serving partitions the model and the KV cache, which adds communication cost but makes larger models servable. For a 70B model (roughly 140GB in FP16), at least two 80GB GPUs are required just for weights, plus additional memory for KV cache and overhead. For the GPU selection criteria, see H100 vs A100.

Managing Memory for Higher Concurrency

Techniques that reduce inference memory are what make high-concurrency serving economical. Quantization shrinks weights from FP16 to INT8 or INT4, reducing the weight memory and allowing larger batches or longer contexts in the same GPU. Paged attention allocates the KV cache dynamically in small blocks rather than reserving worst-case per request, which dramatically improves memory efficiency and raises effective concurrency. Context length capping prevents KV cache from growing beyond what the GPU can hold. Without these techniques, the KV cache's memory consumption limits concurrency to what a naive allocation can support — far fewer requests than the GPU could otherwise serve. For the full memory and latency optimization, see how to reduce LLM inference cost.

Memory consumption components

FAQ

How much GPU memory does LLM inference need?

The total is model weights (approximately 2 bytes per parameter in FP16) plus KV cache (scaling with layers, context length, and concurrency) plus activations. A 7B model in FP16 uses roughly 14GB for weights alone, plus KV cache that can match or exceed that for long-context, high-concurrency workloads. See the estimation method above and the KV cache guide.

Why does GPU memory limit inference concurrency?

Because each concurrent request holds its own KV cache in GPU memory. As concurrency rises, the aggregate KV cache grows linearly, and eventually exhausts the GPU's memory. This is why concurrency is memory-limited in LLM serving — you can serve more requests if each uses less context or if paged attention improves memory efficiency — but there is a hard ceiling set by total GPU memory.

Can I serve a 70B model on a single GPU?

Generally no. A 70B model in FP16 requires roughly 140GB for weights alone, which exceeds any single GPU (H100 and A100 top out at 80GB). Multi-GPU tensor parallelism is required, splitting the model and KV cache across at least two GPUs. For the GPU selection and sizing, see H100 vs A100.

Summary

LLM inference GPU memory is consumed by model weights (fixed, ~2 bytes/param), KV cache (variable, scales with context and concurrency), and activations (batch-dependent). The total determines whether the model fits a single GPU and how many concurrent requests it serves. Estimate memory at the intended context length and concurrency, not in isolation, because the KV cache is often the binding constraint. Use quantization, paged attention, and context capping to manage memory for higher concurrency. For the full serving and optimization framework, see how to reduce LLM inference cost.