Latency Requirements for LLM GPU Sizing and Selection

LLM GPU sizing driven by latency requirements must account for time-to-first-token, time-per-output-token, and the concurrency target — not just whether the model fits in GPU memory. A model that fits may still need more GPUs to meet the latency SLO under real concurrency. For the GPU memory requirements, see how much GPU memory for LLM inference. For the full latency framework, see token generation latency monitoring.

Latency-Driven Sizing Method

Step 1: define the latency SLO — p95 TTFT and p95 TPOT at target concurrency. Step 2: measure per-GPU throughput at the target latency for your model. This is not the GPU's peak throughput — it is the throughput at which latency stays within SLO. Step 3: divide peak concurrency by per-GPU throughput at SLO to get the minimum GPU count for the latency target. Step 4: verify the GPU memory can hold the model weights plus KV cache at the target context length and this concurrency. If memory is insufficient, add GPUs for tensor parallelism. For the memory estimation, see GPU memory for inference.

The common error is sizing for memory fit alone (does the model load?) and neglecting latency (does it serve fast enough?). A model that fits on few GPUs may need more GPUs to meet the latency target under concurrency. Latency-driven sizing answers both questions. For the full sizing method, see how to size AI infrastructure capacity.

FAQ

How do latency requirements affect GPU sizing for LLMs?

Latency sets the minimum GPU count beyond memory fit — the GPUs needed to serve peak concurrency within the TTFT and TPOT SLO. Size for latency and memory; a model that fits may still need more GPUs to meet latency. See the method above.

Summary

Latency-driven LLM GPU sizing accounts for TTFT, TPOT, and concurrency beyond memory fit. For the full framework, see how to size AI infrastructure capacity.