Inference Congestion Control: Admission, Backpressure, and Shedding

Every serving stack eventually faces the moment requests exceed GPUs, and the default response — add a bigger queue — is the one option that never works: it converts overload into latency death while feeling like planning. The real choice is older and harder: block input or drop work. Inference serving inherits that choice in full, because tokens stream under latency contracts that queues violate silently. This page explains the overload truth, maps the three mechanisms that actually manage congestion, and shows how the product's latency contract picks the right one.

The Overload Truth: Queues Move Collapse, Not Fix It

When requests exceed capacity, a queue does not solve the problem — it postpones the collapse while latency dies in line: the classic engineering result holds that an overloaded system must either block input (backpressure) or drop work (shedding), and inference serving inherits the choice fully because tokens stream under latency contracts that queues silently violate.

The classic essay on the subject states the theorem plainly — overloaded systems must choose between blocking on input and dropping work on the floor, because queues only delay the decision while the line grows — and the inference-specific corollary follows from what serving actually sells: a latency contract. A queued request past its deadline is worse than a rejected one, because it consumed queue capacity, saturated attention, and delivered an answer to a client that left. Queueing remains correct for smoothing brief bursts; the truth targets sustained overload, where the queue is the choice to neither block nor drop while violating the contract quietly.

Three Mechanisms, Three Client Experiences

The three mechanisms operate at different layers with different experiences: admission control decides before the queue (rejects arrive fast and honestly), backpressure slows senders cooperatively (clients see their own rate limits), and load shedding drops at the boundary (errors now, latency preserved for the accepted) — and mature serving uses them in combination, with LLM-specific patterns at the API layer including queue-depth limits, retry budgets, and deadline enforcement.

The layer placement is the architecture: admission decisions happen before work enters the queue — rejection, shedding, deadlines, retry budgets — which is precisely where they are cheapest; backpressure operates on senders cooperative enough to receive it; shedding stands at the boundary as the last honest defense. Practitioner coverage of LLM serving documents the API-layer reality of all three — queue depth limits hit under real overload, retry budgets spent, deadlines enforced — and the engineering distinction matters for implementation: backpressure cooperatively slows producers while shedding rejects at the boundary, and mature stacks use both rather than choosing one.

Choosing by the Latency Contract

The product's latency contract picks the mechanism: interactive products serving users shed early and visibly — a fast honest error beats a slow dishonest queue — while batch and internal workloads tolerate backpressure's cooperative slowdown; admission policy encodes the contract (deadlines, retry budgets, priority classes), and the design sin is choosing by infra convenience rather than by what the client experience promises.

• Interactive contracts — shed early: when a user is waiting, the queue is lying to them; reject fast, guide the retry, preserve the experience for the requests you keep.
• Batch and internal contracts — backpressure: senders that can slow should slow — the cooperative mechanism keeps throughput honest without wasted rejection work.
• The admission policy as contract: deadlines enforce what "too late" means, retry budgets prevent retry storms from becoming a second overload, and priority classes decide who survives congestion — the product's promises, compiled into serving behavior.
• Revisit with the contract: when the product's promises change, the congestion design changes with them — the mechanism serves the promise, never the reverse.

The selection rule inverts the usual infra-first instinct: teams reach for queues because queues are easy to add, and the latency contract — the thing the product actually promised — never enters the design. The discipline is to start from the client's side of the experience and work backward to the mechanism, which this site's uptime-engineering and latency pages establish from their own angles: availability work defines what the contract must survive, and this page defines how serving behaves in the moments it cannot.

FAQ

Why not just add a bigger queue?

Because the queue converts overload into latency death: requests wait in line past their usefulness, tokens stream to nobody, and the collapse arrives later but worse — the block-or-drop choice is unavoidable, and the queue is the choice to neither block nor drop while violating the latency contract silently.

Which mechanism should inference serving use?

The combination, placed by layer: admission control before the queue encoding your contract, backpressure to cooperative senders where clients can slow, and shedding at the boundary where they cannot — mature serving uses all three, with the LLM-specific depth limits and retry budgets documented in practitioner coverage.

What do clients experience under shedding versus backpressure?

Shedding: a fast, honest error now, with latency preserved for accepted requests; backpressure: clients slowed to the rate the system sustains; queues: neither — silent latency decay — and the product's promise decides which of those three is the honest experience.