How Does Batching Affect LLM Inference Latency

Quick Answer: Batching raises GPU tokens per second by running several sequences together, and it changes user latency in two directions. It can hide prefill and decode cost across neighbors, and it can add queue wait, padding work, and longer inter-token gaps. The useful question is which percentile you sell, not whether batching is “on.”

Batching’s effect on inference latency is the relationship between scheduler grouping and the delay a single request sees. Static batches wait for a full group. Continuous batching admits and retires sequences while a GPU stays busy. Both change queues. They are not the same mechanism article.

Serving owners should read latency as a curve against batch width and mix, not as a single before-and-after number. This page explains the curve. It does not teach you how to implement continuous batching.

Where does batching add or remove delay?

If you only watch GPU utilization, batching always looks like a win. If you watch user P99, a wide batch at rush hour often looks like a regression. Publish both views or the next incident will be an argument about which graph is real.

Why can tokens per second rise while P99 rises?

The GPU reports work completed. The user reports wait. A wider decode batch completes more tokens per second on the device and can still increase TPOT for each sequence. Prefill batching can raise TTFT if the scheduler holds a short chat prompt until a long RAG prompt arrives.

Mixed context lengths make this worse. One 32k prompt in a group of 1k chats is not “average 8k.” It is a long job with short hostages. Length-aware scheduling is a latency control, not a nicety. So is a cap on how long a request may wait to join a batch.

Static batching shows the wait most clearly because the gate is explicit. Continuous batching hides the gate inside iteration scheduling. The delay is still there when the iteration is packed with heavier sequences. Do not assume continuous batching means “no queue.”

Which knobs move the curve?

Maximum batch width, maximum wait to form a batch, and whether prefill and decode share a queue are the first three knobs. Prefix cache hits shrink prefill and can make a short request less dangerous to its neighbors. KV memory caps shrink the batch you can actually run, which sometimes improves P99 by accident.

Measure with a frozen mix. A synthetic all-short-prompt test will bless a batch size that production RAG traffic will punish. Split the report by prompt-length bucket and by cache hit or miss. A single P99 over all traffic will hide the bucket you will be paged for.

Hardware isolation does not remove batching physics. It only removes noisy neighbors from other tenants. On a dedicated serving pool you still choose a batch policy. One reference to private AI infrastructure is enough: use a stable pool so the curve you measure is your policy, not a stranger’s job.

FAQ

Does a batch size of one always give the best latency?

It often gives the best single-request delay when the GPU is free, and it wastes the GPU when many requests arrive. The product question is the worst delay you will accept at the arrival rate you must take. Batch size one is a point on the curve, not a moral default.

Is this the same as “how continuous batching works”?

No. A mechanism article explains iteration scheduling and in-flight join. This page explains how any grouping policy moves queue, TTFT, and TPOT. You can understand the mechanism and still set a harmful max batch if you never plot user percentiles.

How should we talk about batching in an SLO?

State the mix, the percentile, and whether the clock is TTFT, TPOT, or end-to-end. “Keep batching on” is not an SLO. “P95 TTFT under X on the production mix” can survive a runtime change. Attach the batch caps that were in force during the baseline.

Can we batch training and inference together to save GPUs?

Not as a latency strategy. Training steps and serving batches do not share a user clock. If they share a GPU, inference latency becomes a function of someone else’s step. That is a partition problem, not a batch-size problem.

Why did latency get worse after we “improved utilization”?

Because utilization counts busy SMs, not user wait. The improvement was likely a wider batch or a longer queue. Plot P99 against the change. If P99 moved the wrong way, the utilization win was a transfer of delay from the GPU idle account to the user.

Summary

Batching affects inference latency by trading device efficiency for queueing, neighbor prefill, and wider decode steps. Tokens per second can rise while user P99 rises. Plot the curve on a frozen traffic mix and own the percentile you sell. Mechanism details belong in a separate article.

When measurements must not include another tenant’s jobs, use a dedicated serving path such as private AI infrastructure and still treat batch policy as your latency control.