Throughput and Latency Cost Tradeoff in LLM Serving

The throughput-latency cost tradeoff in LLM serving is the balancing act between serving more tokens per GPU-hour (lower cost per token) and keeping per-request latency within target — and the operating point is where the batch size, concurrency, and GPU count meet the latency SLO at the lowest cost. For the batching mechanics, see LLM inference batching. For cost estimation, see how to estimate LLM serving cost.

Understanding the Tradeoff

Higher throughput → lower cost per token, higher latency: larger batches reuse weight movement across more requests, producing more tokens per forward pass. Cost per token drops because the same GPU cost is spread across more tokens. But each request waits longer for its token because it shares the GPU with more peers. Lower latency → higher cost per token, lower throughput: smaller batches serve each request faster because the GPU is shared with fewer peers. But throughput drops because fewer tokens are produced per forward pass, so cost per token rises. The sweet spot: the largest batch size that keeps p95 latency within the SLO. At this point, you capture the maximum throughput the latency target allows, producing the lowest cost per token that meets the user's experience requirement. For the latency monitoring, see token generation latency monitoring.

Finding the Operating Point

Define the latency SLO (p95 TTFT and TPOT). Measure throughput and cost per token across a range of batch sizes under production traffic. The cost-optimal operating point is the largest batch size where p95 latency stays within the SLO. Larger than that breaks the SLO; smaller than that leaves throughput — and cost savings — on the table. For the GPU-level cost, see GPU cost per hour vs TCO.

FAQ

How do I find the right batch size for cost and latency?

Define the latency SLO, then measure throughput and cost per token across batch sizes. The optimal is the largest batch that keeps p95 latency within the SLO. This captures maximum throughput at the latency target. See the method above.

Summary

The throughput-latency tradeoff in LLM serving is optimized by setting the batch size to the largest value that meets the latency SLO. For the full framework, see LLM inference batching.