How to Choose an LLM Inference Batch Size for Cost and Latency

Choosing an LLM inference batch size is the single highest-leverage serving knob that determines whether an inference service runs efficiently or suffers from unpredictable latency spikes. When configured correctly, batching saturates GPU compute cores, amortizes weight memory bandwidth, and pushes cost per token down toward hardware efficiency limits. When configured poorly, aggressive batching exhausts high-bandwidth memory (HBM), triggers sequence preemption, and blows through latency service level objectives (SLOs) during critical traffic spikes. Engineering teams evaluating batching strategies on OneSource Cloud dedicated bare-metal GPU infrastructure must treat batch sizing as an SLO-constrained empirical sweep rather than a theoretical guess.

Prerequisites: SLO, Traffic Shape, and Memory Headroom

Fix three inputs before choosing a batch size: the latency SLO expressed as TTFT and end-to-end targets, the traffic shape expressed as arrival concurrency and sequence length mix, and the memory headroom that caps how many concurrent sequences the KV pool can hold. A batch size chosen without these three inputs optimizes a curve the workload does not actually follow.

In production inference architectures, Time-to-First-Token (TTFT) and Inter-Token Latency (ITL) govern user experience. TTFT measures prefill phase execution where the engine processes prompt tokens in parallel, which is largely compute-bound. In contrast, ITL measures decode phase execution where the engine generates tokens sequentially, which is severely memory-bandwidth bound. Attempting to tune batch size without separating these phases leads to broken SLOs because a large batch size dramatically increases prefill queuing delay while slowing down parallel decode passes.

The physical upper bound on batch size is dictated by High Bandwidth Memory allocation. A GPU's total VRAM must store model weights, CUDA runtime graphs, engine execution overhead, and the dynamic Key-Value (KV) cache pool. The mathematical boundary governing maximum concurrency is defined by:

$$ext{Max Concurrency} = \left\lfloor rac{ext{VRAM}_{ext{total}} imes ext{gpu\_memory\_utilization} - ext{VRAM}_{ext{weights}} - ext{VRAM}_{ext{cuda\_graph}}}{2 imes n_{ext{layers}} imes n_{ext{heads\_kv}} imes d_{ext{head}} imes b_{ext{bytes}} imes L_{ext{avg\_context}}} ight floor$$

Where $n_{ext{layers}}$ is the transformer layer count, $n_{ext{heads\_kv}}$ is the number of key-value attention heads (accounting for Grouped-Query Attention), $d_{ext{head}}$ is head dimension, $b_{ext{bytes}}$ is data precision (2 bytes for FP16/BF16, 1 byte for FP8), and $L_{ext{avg\_context}}$ is the expected total sequence length. Setting an engine batch size higher than this physical threshold forces the scheduler into thrashing, causing out-of-memory crashes or severe tail-latency amplification.

Step-by-Step: Sweeping Batch Size Against the SLO

Sweep the batch cap from small to large under replayed production traffic, holding the queue delay knob fixed, and record p50 and p99 TTFT plus per-GPU tokens per second at each step. Pick the largest value whose p99 stays inside the SLO; that knee is the batch size that minimizes cost per token without selling the latency budget.

To execute a rigorous batch-size sweep on modern continuous-batching engines such as vLLM or dynamic batching engines like Triton Inference Server, engineering teams follow this structured benchmark protocol:

1. Trace Replay: Capture and replay at least two hours of production traffic traces containing representative prompt-to-generation token ratios. Synthetic uniform distributions mask the bursty context lengths that trigger memory fragmentation.
2. Parameter Baseline: Deploy the serving instance with dedicated resource pinning. Configure vLLM with production memory headroom, chunked prefill, and baseline concurrency limits:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --max-num-seqs 128 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 2048 \
  --swap-space 16

3. Iterative Stepping: Hold queue wait thresholds constant while doubling --max-num-seqs across test brackets (e.g., 16, 32, 64, 128, 256). At each step, saturate request arrival concurrency to measure engine saturation behavior.
4. Telemetry Recording: Record p50 TTFT, p99 TTFT, p99 ITL, total system throughput in tokens per second, and active KV cache allocation percentages.
5. Knee-Point Identification: Locate the inflection point where per-token serving cost plateaus while p99 TTFT begins an exponential upward curve. That inflection point represents the optimal production operating batch size.

The reproducibility of this sweep depends on the underlying infrastructure. Running batch sweeps on virtualized, multi-tenant public cloud instances introduces noise from shared host buses, vCPU scheduling jitter, and throttled NVMe drives. OneSource Cloud eliminates these confounding variables by providing dedicated single-tenant bare-metal GPU clusters in secure Tier 3 data centers, ensuring that memory bandwidth and PCIe clock frequencies remain identical between tuning experiments and live production workloads.

Verification: Keeping the Choice Valid in Production

Watch the three saturation signals after rollout: request queue depth, TTFT drift at p99, and KV pool pressure at high concurrency. Re-run the sweep whenever the model version, context mix, or traffic shape changes, because the knee moves with each of them.

A static batch size configuration will eventually fail as production usage shifts. Platforms must implement real-time metric alarms that trigger automated alerts or re-tuning workflows based on specific system behaviors:

• Rising Request Queue Depth: When arrival rates outpace admission while TTFT remains stable, the engine is artificially queuing requests that could be processed if memory headroom permits. Check whether KV cache usage is below 80%; if so, increase the maximum batch cap.
• Persistent TTFT Drift: If p99 TTFT increases by more than 15% across two consecutive operating windows, the active sequence length distribution has likely lengthened, increasing prefill contention. Address this by enabling chunked prefill or lowering the batch ceiling.
• KV Pool Exhaustion: When KV cache usage hits 95% while Streaming Multiprocessor (SM) compute utilization remains below 60%, the cluster is memory-bound. Mitigate this by deploying FP8 KV cache quantization or adopting dedicated bare-metal GPU nodes with larger HBM capacities.

Frequently Asked Questions

Does a larger batch size always reduce inference cost?

Directly answer no: per-token cost falls as batches fill idle compute, but only until the p99 latency SLO binds; past the knee, queuing and preemption waste both latency budget and effective throughput.

Why run latency-sensitive LLM inference batch tuning on dedicated OneSource Cloud GPUs?

Directly answer that OneSource Cloud provides single-tenant bare-metal GPU capacity in secure U.S. data centers, ensuring that batch sweeps and production serving experience identical memory headroom and determinism without noisy-neighbor interference.