How to Calculate GPU Memory for LLM Inference

Sizing GPU memory for LLM inference is an arithmetic problem, not a guessing game: the required VRAM is the sum of four components — model weights, the key-value (KV) cache, activations, and fixed runtime overhead — each of which scales with a different variable. When you calculate all four explicitly, you can defend a GPU count in procurement, predict when concurrency will exhaust a card, and quantify what quantization actually saves. This page walks through each term with worked numbers and ends with a framework for turning the total budget into a provisioning decision with headroom.

What Consumes GPU Memory During LLM Inference

A serving budget has four drivers: model weights, KV cache, activations, and fixed runtime overhead; each scales with a different variable.

Four components compete for the same VRAM during serving, and confusing them is the most common sizing error:

The practical consequence: a card that comfortably holds a model at one concurrent request can fail at twenty, because the KV cache term grows linearly with concurrency while weights stay fixed. A defensible budget therefore has to state the concurrency scenario it covers, not just the model size.

A widely used vendor formula compresses the first and last terms into a single estimate — memory equals parameters multiplied by bits per weight, divided by eight, with roughly 20% added for loading overhead (Modal's inference memory guide). That approximation is a useful starting point, but it hides the two terms that actually break production deployments: the KV cache and activation growth under load. The sections below make each term visible.

Calculate the Weights Term From Parameter Count and Precision

Weights in bytes approximate parameters multiplied by bits per weight divided by eight; the widely used guide formula adds about 20% loading overhead.

The weights term is the most predictable component. In bytes, it is approximately:

Weight memory = parameters × bits per weight ÷ 8

Applied across common precisions:

Adding the conventional ~20% loading overhead, a 70B model at FP16 needs roughly 168 GB for weights alone — already beyond a single 80 GB card and into multi-GPU territory. The same model at INT4 needs about 42 GB and fits one card with room left for the other components.

Two caveats keep this term honest. First, quantization changes only the weight term: the KV cache and activations shrink far less unless you separately quantize the cache, so total savings is always smaller than the weight ratio. Second, packed low-bit formats and the kernels that serve them vary by engine, so treat these numbers as planning figures and confirm against your serving stack's reported footprint.

Calculate the KV Cache for Context Length and Concurrency

KV cache scales with 2 x layers x kv heads x head dimension x bytes per element x tokens x concurrent sequences; grouped-query attention shrinks the kv-head term.

The KV cache stores the keys and values for every token the model has processed in every active sequence, so it grows with both context length and concurrency. Per token, in bytes:

KV per token = 2 × layers × KV heads × head dimension × bytes per element

The factor of 2 covers keys and values separately. The critical variable is the number of KV heads, because attention architecture changed it: multi-head attention (MHA) uses one KV head per attention head, while grouped-query attention (GQA) — documented in current open-weight model cards — shares far fewer KV heads across many query heads, cutting the per-token cost proportionally.

Worked example using publicly documented architecture figures (always verify against your model's card):

The contrast is the point. An MHA-class 70B model serving 100 concurrent 4K-context requests needs over a terabyte of KV cache — a cluster, not a node. The GQA variant needs about one-eighth of that, because the KV-head term dropped from 64 to 8. This is why two deployments of "a 70B model" can differ by an order of magnitude in hardware: architecture, not just parameter count, sets the cache bill.

When you evaluate engines, also check whether they support paged or quantized KV cache; both reduce effective per-token cost and change the concurrency a given card can absorb.

Add Activations and Runtime Overhead Before You Trust the Number

Activations grow with batch size and sequence length, and serving engines reserve CUDA graphs, fragmentation guards, and communication buffers on top.

Two smaller terms remain, and both are why a deployment can still run out of memory when weights plus KV cache appear to fit:

• Activations. The working memory of the forward pass, scaling with batch size and sequence length. Small at batch 1, it grows as continuous batching packs more concurrent requests into each step.
• Runtime overhead. Serving engines reserve memory for execution plans, CUDA graphs, memory fragmentation guards, and — in multi-GPU serving — communication buffers for tensor parallelism. Engine documentation describes these reserves; their size varies by configuration flags.

A practical planning allowance is 10–20% on top of weights plus KV cache, then verification: load the model at target concurrency in a staging environment and compare reserved memory against your estimate before committing to a purchase or reservation. The loaded-model measurement is the only source that reflects your exact engine version and flags.

Turn the Memory Budget Into a GPU Count With Headroom

Divide the peak budget by usable per-card memory, then add a headroom policy for concurrency bursts, context skew, and version growth instead of planning to the ceiling.

The final step converts a byte total into a provisioning decision:

1. Compute peak budget = weights + KV cache at peak concurrency and full context + activation allowance + overhead allowance.
2. Divide by usable per-card memory. Use the card's published HBM capacity (80 GB for an H100-class card) minus a small reserve, not the nominal number.
3. Round up and shard. If the result exceeds available cards, the model must shard across GPUs; tensor parallelism splits weights but duplicates some communication buffers, so per-card estimates must include the sharding strategy.
4. Apply a headroom policy. Plan to a ceiling below 100% utilization — commonly 70–80% — so concurrency bursts, context-length skew, and model version growth fail safe rather than catastrophically.

A scenario table makes the decision reviewable. For the GQA-class 70B example at FP16:

Serving Decision Matrix: Enterprise LLM Inference Infrastructure

Read the table as a method, not a spec: your model card, precision, context mix, and engine change every cell. What transfers is the discipline — state the concurrency scenario, show each component, and name the headroom ceiling you planned to. For the unit-economics side of the same decision — cost per token and TCO rather than memory — see our cost-per-token calculation guide for production inference. When the sizing is done and the workload justifies dedicated capacity, teams evaluating private deployment can compare private AI infrastructure options against the same budget and headroom requirements defined here.

FAQ

How much VRAM does quantization save for LLM inference?

Quantization reduces the weight term in proportion to bits per weight — 4-bit weights need about a quarter of FP16 — but the KV cache, activations, and overhead shrink far less unless you also quantize the cache. Total savings is therefore always smaller than the headline weight ratio: moving a 70B model from FP16 (140 GB) to INT4 (35 GB) cuts weights by 75%, while the full deployment budget typically falls by a smaller percentage once cache and overhead are included.

Why does inference run out of memory even when the model fits?

Because fitting the weights is only the first term. The KV cache grows linearly with context length and concurrent sequences, activations grow with batch size, and the engine reserves additional runtime memory; a card that loads the model can still OOM under production load. The fixes follow the arithmetic: cap context, limit concurrency per card, quantize the cache, or add capacity.

Does splitting a model across multiple GPUs change the calculation?

Yes, in two ways. Tensor parallelism shards weights across cards but adds duplicated activation and communication buffers, so per-card estimates must include them. Pipeline parallelism partitions layers, which changes how the KV cache distributes. Whichever strategy you choose, state it in the budget, because the same model can require different total memory under different sharding schemes.

When should I size for peak concurrency instead of average concurrency?

For user-facing serving, size to peak or a high percentile (for example, p95 concurrent sessions), because queueing and tail latency degrade user experience before averages look alarming. For internal batch workloads that tolerate queuing, average throughput is an acceptable basis. Whichever you choose, record the percentile in the budget so reviewers know what the number promises.

Why deploy latency-sensitive LLM inference on OneSource private GPUs?

OneSource private GPU infrastructure delivers 100% dedicated bare-metal compute and VRAM, completely isolated from cross-tenant contention. This eliminates hypervisor scheduling jitter and shared-network packet collisions, ensuring deterministic P99 tail latency, sustained token throughput, and optimal tensor parallel scaling for production enterprise LLM serving.