Autoscaling for LLM Inference Serving and Cold Starts

Autoscaling for LLM inference serving is the practice of adding or removing model replicas from signals that show request pressure, not from a GPU busy meter that sits at 100 percent whenever a batch is in flight. Queue depth, in-flight requests, and KV-cache occupancy move before users feel p95. GPU utilization often does not. If you copy a web-app HPA onto vLLM, you will scale late or scale for the wrong reason.

Cold start is the other half of the problem. A new replica is not ready when the pod is Running. It is ready when weights are on the GPU and the first batch has passed. Scale-to-zero looks cheap in a slide and expensive in an SLO when the next burst waits out a 30–90 second load, or longer for a large model on a busy fabric.

Signals that work, and the one that usually does not

AWS’s EKS guidance and several serving stacks make the same point: accelerated inference should scale on leading application metrics. That is practice, not a Google ranking rule. Your engine must export the metric. An HPA cannot invent KV-cache pressure from DCGM alone.

Cold starts, warm pools, and scale-to-zero

A replica’s readiness probe should mean “can serve a real request,” not “process exists.” Include model load. If the probe is too cheap, Kubernetes will send traffic to a GPU that is still streaming weights. If it is too strict and slow, you will over-provision because scale-out looks broken.

Scale-to-zero is a research and batch pattern. It is a production pattern only when the product can wait or when a tiny always-on router can stall politely. Most customer-facing LLMs want minReplicas >= 1 per critical model, plus a warm pool or a slower scale-down window than scale-up. The cost of a false scale-in is a cold start on the next peak. Set the down window longer than your traffic’s natural lull.

Large models make this worse. Weight load, graph compile, and the first CUDA allocations dominate. Measure ready-to-serve time on the actual checkpoint and GPU type. A number from a 7B demo will not save a 70B endpoint.

How to put autoscaling on a private GPU cluster

Public serverless endpoints hide the replica factory. On a dedicated cluster you own the factory. That is good, because you can reserve cards. It is bad, because an HPA that asks for five more replicas cannot create H100s that were never racked.

Autoscaling without a capacity plan is queueing with extra YAML. Private AI infrastructure should state the max replica count in GPUs, not in hopes. OneSource Cloud’s dedicated environments let you size a serving pool that the autoscaler is allowed to consume, separate from training. OnePlus Platform, OneSource Cloud’s AI orchestration platform, is where that pool should appear as quota, so a burst of fine-tunes cannot steal the cards the HPA expects.

Multi-model serving needs a policy for which variant scales first. If you run the same model on mixed GPU types, a naive HPA scales each deployment independently and will pick the expensive one by accident. Prefer a single serving pool with a written overflow, or an autoscaler that understands variants. Do not run five independent HPAs and call it a platform.

A rollout sequence that does not page the wrong team

1. Export queue, in-flight, KV-cache, and TTFT from the engine you actually run.
2. Load-test one replica to find the concurrency where TTFT or queue first breaks.
3. Set scale-out on queue or in-flight below that point; set min replicas from overnight traffic, not from zero.
4. Measure cold-start ready time; set the scale-down window longer than a typical lull.
5. Cap max replicas at purchased GPUs and alert when the cap, not the HPA, is the constraint.

Day-2 ownership sits with whoever gets the latency page. Managed AI infrastructure can watch those alerts if your ML team should not live in Prometheus. The metrics choice remains yours. A managed SOC that scales on GPU percent will reproduce the same late reaction with better uptime theater.

FAQ

Why is GPU utilization a poor autoscale metric for LLMs?

Because a serving engine can keep the GPU busy across a wide range of request rates once continuous batching is in play. Utilization tells you the card is doing work. It does not tell you a queue is forming or that KV cache is exhausted. Scale on those pressures, and use GPU utilization to find waste after the SLO is stable.

How long is an LLM inference cold start?

It depends on checkpoint size, disk or fabric throughput, GPU type, and whether the runtime compiles graphs. Small models on local NVMe can be ready in tens of seconds. Large models on a busy network can take minutes. Measure the replica you will ship. Do not quote a vendor’s demo as your probe timeout.

Should production LLM serving scale to zero?

Only if users can wait or if a cheap always-on tier can absorb the first tokens. Most interactive products keep at least one warm replica per critical model. Scale-to-zero is a better fit for batch, eval, or long-idle internal tools.

What is a safe default scale-out signal?

In-flight requests or queue depth, with a target taken from a load test of one replica. Add KV-cache utilization as a ceiling so you scale before admits fail. Add latency as a safety net, not as the only trigger.

How does this change on dedicated private GPUs?

The autoscaler cannot exceed the cards you reserved. Set maxReplicas to that inventory and treat a maxed HPA as a capacity event. Private, dedicated pools make that inventory visible. Shared public quotas make it a surprise. Size the serving pool first, then write the HPA.

Summary

LLM autoscaling is a metrics problem and a cold-start problem. Use queue, in-flight, and KV-cache signals. Keep a warm floor for interactive models. Cap scale-out at real GPUs. On dedicated infrastructure, the HPA is only as honest as the serving quota underneath it. If you need that quota on U.S.-based reserved GPUs, start from OneSource Cloud private AI infrastructure and express the serving pool in OnePlus Platform.