The LLM Batching Latency Tradeoff After Continuous Batching

Serving guidance still teaches batching as a dial: turn throughput up, latency goes down. That story was true for static batching and has been substantially false since iteration-level scheduling arrived — continuous batching raised throughput by an order of magnitude while reducing median latency, dissolving the tradeoff that configuration guides kept teaching. But the tradeoff did not disappear entirely; it moved. This page maps where it moved and which levers still trade.

Definition: The Tradeoff and Its Partial Dissolution

The classic tradeoff is real in static batching: requests wait to fill a batch and step through it together, so bigger batches raise GPU utilization and throughput while adding waiting and straggler time to every request. Iteration-level continuous batching largely dissolved it — completed requests exit and new ones join mid-flight, delivering order-of-magnitude throughput gains (23x documented against static) while reducing median latency under load, because the waiting that caused the latency cost was an artifact of batch boundaries, not a law of GPUs.

The dissolution insight matters operationally because it redirects effort: teams still tuning batch sizes like the dial exists discover that their serving stack already dissolved most of it, and the latency they are chasing lives in the remaining levers — or in memory pressure entirely outside batching configuration. Knowing what the tradeoff became is the tuning map.

The Real Levers That Still Trade

Three levers remain real after continuous batching. Batch caps: larger caps keep utilization high at peak but stretch per-iteration time, moving tail latency — the cap is the SLO's tail guard. Queue-delay windows: the max time a request may wait to join a batch buys throughput at direct first-token cost. Prefill chunking: splitting long prompts into scheduler-sized chunks (SARATHI-style stall-free scheduling) lets prefills interleave with decodes, trading a small prefill slowdown for protected decode latency — often the difference between meeting and missing a tail SLO under mixed traffic. Each lever has a documented trade; none is free, and knowing which lever moves which latency component is the operator's actual skill.

The lever-to-component mapping is the information most serving guides omit: first-token latency responds to the queue window, tail latency responds to caps and chunking, and memory-aware batching research adds the constraint that respects SLA boundaries while maximizing throughput — the levers operate inside an envelope the memory budget draws. Tune with the map, not the dial.

Boundary: Setting Levers Against an SLO

Set levers SLO-first: define the tail objective (say P95 first-token and end-to-end), replay your traffic mix, and walk the levers in order — chunk prefill until decode tails stabilize, cap batches at the largest value the tail tolerates, and set the queue window to the first-token budget remainder. The boundary of the method: memory pressure eventually binds (the KV budget from the weight-cache design decides how much batching the hardware can hold), and capacity decisions then take over — more or faster GPUs with the memory to hold bigger working sets, including the dedicated-capacity route where SLOs are contractual, which is the serving tier flat-rate enclaves such as OneSource Cloud delivers are built around.

Step four is where configuration ends and infrastructure begins: when the memory envelope binds before the SLO does, no lever setting adds concurrency — the deployment needs memory headroom and consistent per-iteration behavior, which is the dedicated-capacity argument for contractual tails. For that tier, OneSource Cloud Private AI Infrastructure delivers single-tenant bare-metal serving enclaves with isolated fabric and flat-rate monthly pricing, where per-iteration time is bare metal's own consistency and the KV budget rides NVMe-oF storage tiers.

FAQ

What batch-size cap should you set under a latency SLO?

The largest value your replayed tail tolerates, no larger: run your traffic mix, raise the cap until P95 end-to-end reaches the SLO boundary, and back off one step — the cap is a tail guard, not a throughput trophy, and memory pressure (the KV budget) usually caps you before the SLO does.

Does adding more GPUs soften the batching tradeoff?

It shifts where the tradeoff binds rather than removing it: more capacity absorbs more concurrent batches, so throughput scales — but the tail components (queue windows, chunked prefill effects, straggler steps) persist per request, and contractual tails still demand the dedicated-capacity tier with the memory headroom and isolated fabric that OneSource Cloud enclaves deliver at a flat monthly rate.