GPU Requirements for LLM Inference: Memory, Throughput, and Sizing

GPU requirements for LLM inference are determined by four interacting factors: the model size that sets baseline memory, the context length that grows memory with each request, the concurrency target that multiplies memory needs, and the latency or throughput goal that determines GPU type and count. Sizing inference GPUs is a system problem, not a single-number lookup.

For teams deploying large language models, GPU sizing is the decision that most shapes both cost and performance. Undersize and the deployment cannot meet latency or concurrency targets; oversize and the organization pays for idle capacity. The common mistake is sizing from the model alone, which accounts for the weights but ignores the memory that active requests consume and the throughput the serving stack must deliver. Understanding the full set of requirements helps teams size inference capacity defensibly rather than by trial and error.

The Four Factors That Determine Inference GPU Requirements

Inference GPU requirements are the product of four factors that interact. Changing any one shifts the total requirement, which is why sizing must consider all of them together. The table below maps the factors and how each drives GPU needs.

Model Size: The Memory Baseline

The model's parameter count and the precision it runs at set the baseline GPU memory required just to hold the weights. A model with tens of billions of parameters requires substantial memory before it serves a single request, and larger models may require multiple GPUs just to hold their weights through techniques such as tensor parallelism. This baseline is the floor below which the deployment cannot operate regardless of other factors.

Quantization reduces this baseline by lowering precision, which shrinks the memory footprint and can raise throughput with modest quality impact. Many production deployments use quantized models specifically to fit larger models into available GPU memory. The model size decision and the quantization decision are linked, because together they determine the memory floor the hardware must meet.

Context Length and the KV Cache

Beyond the weights, each active request consumes memory for its key-value cache, which stores the intermediate state needed to generate tokens efficiently. The KV cache grows with context length, so a request with a long prompt or an extended conversation consumes more memory than a short one. This is why context length is a GPU requirement factor in its own right.

The practical consequence is that the same model can serve very different numbers of concurrent requests depending on context length. A deployment handling short prompts may serve many requests per GPU, while the same deployment handling long documents may serve far fewer. Teams must size using their actual context-length distribution, not an average, because long-context requests disproportionately consume memory and define the capacity ceiling.

Concurrency Multiplies Memory

Concurrency multiplies the per-request memory demand. Serving ten simultaneous requests consumes roughly ten times the per-request KV cache memory, plus the shared weights. This is why concurrency target is the factor that most often drives GPU count upward, because each additional concurrent request adds memory pressure until the GPU is full.

The interaction between context length and concurrency is what makes inference sizing non-obvious. A deployment sized for high concurrency with short contexts may collapse when contexts lengthen, because the memory per request grows while the GPU capacity stays fixed. Sizing must account for the realistic combination of the two, not each in isolation.

Latency and Throughput Targets

Latency and throughput targets determine which GPU type and how many are needed to meet the performance the deployment promises. Latency, how fast a single response returns, depends on GPU speed and memory bandwidth. Throughput, how many tokens per second the deployment generates overall, depends on GPU count and utilization efficiency. A deployment can be latency-optimized, throughput-optimized, or balanced, and each implies different hardware.

Faster GPUs such as H100 accelerators deliver lower latency and higher throughput per device than older hardware, but at higher cost. The relevant metric is tokens served per dollar at the target latency, which depends on how well the serving software uses the hardware. Teams should measure throughput in their actual serving stack rather than trusting synthetic benchmarks, because real performance depends heavily on batching and scheduling configuration.

Batching and Utilization

Batching is the lever that turns raw GPU capability into throughput. Continuous batching techniques process incoming requests together dynamically, keeping GPUs busy and raising tokens served per device. Without effective batching, GPUs idle between requests and the deployment needs more hardware to meet the same throughput target. This is why the serving software, not just the GPU specifications, is part of the requirement.

Utilization matters because idle GPUs cost the same as busy ones. A deployment with poor utilization needs more GPUs to meet its target, which raises cost without proportional benefit. Teams should treat the serving architecture and its achievable utilization as part of the GPU requirement, because a well-utilized smaller fleet can outperform a poorly-utilized larger one.

A Practical Sizing Method for LLM Inference

Sizing inference GPU capacity means working through the four factors in sequence to arrive at a defensible hardware plan. The goal is a configuration that meets the performance target without excess capacity, supported by stated assumptions rather than guesswork.

First, determine the model and its precision, which sets the memory baseline for weights. Second, characterize the workload's context-length distribution and concurrency target, which sets the KV cache memory demand. Third, add weights and cache memory to find the per-GPU concurrency ceiling, then divide the target concurrency by that ceiling to estimate GPU count. Fourth, check the GPU type and count against the latency and throughput targets using measured benchmarks, and adjust. Fifth, apply a utilization factor to account for serving inefficiency, because real deployments never reach theoretical limits.

Planning for Growth

Inference demand tends to grow as users adopt a deployment, so sizing should include headroom or a clear expansion path. GPU procurement lead times can be long, and running out of capacity mid-growth forces either degraded service or emergency purchasing at premium rates. Building in headroom, or choosing a provider that can expand capacity on a known timeline, protects the deployment as usage rises.

Choosing Infrastructure for LLM Inference

Once the GPU requirements are sized, the infrastructure choice determines how those requirements are met. Enterprises face the same broad options as other AI workloads: public cloud GPU services for flexibility, dedicated GPU infrastructure for predictability and control, and managed providers that supply and operate the environment.

For production inference that must be reliable and cost-predictable, dedicated GPU infrastructure with managed operations is often the strongest fit, because it removes the quota and pricing volatility of shared cloud while supplying the operations capability that inference demands. Providers such as OneSource Cloud that offer private AI infrastructure with managed operations help teams meet inference GPU requirements without building a full operations function in-house.

FAQ

How much GPU memory do I need for LLM inference?

You need enough memory to hold the model weights plus the KV cache for all concurrent requests at your target context length. The weights set the baseline, and the cache grows with both context length and concurrency. Sizing from weights alone understates the requirement, because active requests add substantial memory on top.

How does context length affect GPU requirements?

Longer context grows the KV cache per request, which consumes more memory and reduces how many concurrent requests a GPU can handle. A deployment sized for short prompts may serve many requests per GPU, while the same deployment handling long documents serves far fewer. Size using your actual context-length distribution, not an average.

Which GPU is best for LLM inference?

The best GPU depends on your latency and throughput targets relative to cost, measured as tokens served per dollar. Faster GPUs such as H100 accelerators deliver lower latency and higher throughput per device but cost more. The relevant comparison is real-world performance in your serving stack, not synthetic benchmarks.

How many GPUs do I need to serve an LLM?

Divide your target concurrency by the per-GPU concurrency ceiling, which is the memory available for KV cache after weights divided by the per-request cache size at your context length. Then check the result against your throughput target and apply a utilization factor for serving inefficiency. The answer depends on all four requirement factors together.

Can I reduce GPU requirements through quantization?

Yes. Quantization lowers the precision of the model, which shrinks the weight memory footprint and can raise throughput with modest quality impact. Many production deployments use quantized models to fit larger models into available GPU memory. The model size and quantization decisions are linked because together they set the memory floor.

Summary

GPU requirements for LLM inference are determined by model size, context length, concurrency, and latency or throughput targets working together. Sizing from the model alone understates the requirement, because active requests add KV cache memory that grows with context length and multiplies with concurrency. A defensible sizing method works through all four factors in sequence and applies a utilization factor for real-world serving inefficiency.

For production inference that must be reliable and cost-predictable, dedicated GPU infrastructure with managed operations is often the strongest fit. OneSource Cloud's private AI infrastructure and managed AI infrastructure services are designed to meet inference GPU requirements without forcing teams to build a full operations capability in-house.