What Causes High P95 Latency When Serving LLMs

High p95 latency in LLM serving typically comes from one of five causes — queue contention, KV cache pressure, large prompts, batch size effects, or resource saturation — and each has a different fix, which is why diagnosis must precede action. Treating a queue problem by adding memory, or a KV cache problem by adding GPUs, wastes resources without fixing the latency, because the fix must match the cause. For the full latency monitoring framework, see token generation latency monitoring.

For production LLM serving teams, p95 latency is the metric that reflects what users actually experience. Average latency hides the slow requests; the tail exposes them. When p95 latency rises, a fraction of users — one in twenty — is experiencing slow responses, and diagnosing why quickly is the difference between a brief degradation and a sustained outage. This guide explains the five causes, how to diagnose which one is active, and what to do about each.

Why P95 Latency, Not Average, Catches Problems

Average latency is a misleading metric for LLM serving because it is pulled down by the many fast requests and hides the slow ones. A system with a 200ms average and a 3-second p99 has a serious problem that the average conceals — one in a hundred requests takes fifteen times longer than typical. Users hit those slow requests, and their experience defines the service's perceived quality. For the full treatment of why tails matter, see our token generation latency monitoring guide.

The gap between median and p95 is itself a diagnostic signal. A small gap means latency is consistent; the system is healthy. A large and growing gap means some requests are hitting a contention point others avoid, which is the signature of an emerging problem. Tracking the median-to-p95 spread over time catches problems before the p95 itself crosses a threshold.

Cause 1: Queue Contention

Queue contention raises p95 latency when requests arrive faster than the system can process them, building a queue. The requests at the back of the queue wait the longest, producing a long tail. The diagnostic signature is a rising queue depth correlated with rising p95 latency, with a stable time-to-first-token for the requests that do get processed — meaning the serving is fine once a request starts, but waiting to start is what hurts.

The fix depends on the contention pattern. If queue depth is rising due to a traffic spike, autoscaling capacity (or rate limiting to cap arrivals) addresses the cause. For how to manage queues and rate limiting, see how to prevent inference queue overload. If queue depth is rising due to capacity loss (a node failure), restoring capacity is the fix.

Cause 2: KV Cache Pressure

KV cache pressure raises p95 latency when the cache consumes enough GPU memory that the system spends time on memory management — eviction, re-computation, or fragmentation handling — rather than on generation. The diagnostic signature is a rising p95 latency correlated with high KV cache occupancy and possibly a rising inter-token latency (tokens per second dropping), because memory pressure slows each decode step.

The fix is to reduce KV cache pressure: cap the maximum context length to what users actually need, enable paged attention to make cache allocation efficient, evict idle cache aggressively, and if necessary add GPU memory or reduce concurrency. For the mechanics of the KV cache and how it drives memory and latency, see our KV cache explainer.

Cause 3: Large or Variable-Length Prompts

Large prompts raise the time-to-first-token (TTFT) because the prefill phase — processing the prompt and building the initial KV cache — is compute-heavy and scales with prompt length. If the workload has a mix of short and long prompts, the long prompts produce a latency tail. The diagnostic signature is a rising p95 TTFT correlated with large prompt lengths in the tail requests.

The fix is a combination of prompt caps (limit maximum length to what the use case actually needs), prompt routing (send long prompts to a dedicated model or replica), and prefill optimization. Prompt length variation is inherent in many applications — users ask different-length questions — so the fix is often architectural (separating long-prompt serving from short) rather than a configuration change.

Cause 4: Batch Size and Scheduling Effects

Batch size affects p95 latency because larger batches raise per-request latency — each request shares the GPU with more peers. If the batch size grows under high load, p95 latency rises even though throughput is higher. The diagnostic signature is a correlation between batch size and p95 latency, often with throughput also rising — meaning there is a tradeoff, not a defect. For the batching mechanics, see LLM inference batching and how continuous batching works.

The fix is to cap the batch size at the workload's latency target, trading some throughput for tail stability. The right batch size is the largest one that keeps p95 latency within target — which is workload-specific and must be tuned empirically.

Cause 5: Resource Saturation

Resource saturation — GPU compute, memory bandwidth, or memory capacity — raises p95 latency across the board when the system is running near its limit. The diagnostic signature is uniformly high utilization paired with rising latency, with no single outlier cause. This is the simplest to diagnose (utilization is near peak) and the most expensive to fix (add capacity or reduce load).

The fix is adding capacity, reducing concurrency, or using optimizations — quantization, model routing to smaller models for simple requests — that lower the resource demand per request. For the cost optimization framework, see how to reduce LLM inference cost.

Diagnosing p95 latency causes

FAQ

What causes high p95 latency in LLM serving?

Five causes: queue contention (requests waiting), KV cache pressure (memory management overhead), large prompts (long prefill times), batch size (too large for latency target), and resource saturation (system near limit). Each has a different diagnostic signature — the correlation between the metric and the latency rise identifies which cause is active — and a different fix. Diagnose before acting. See the full table above.

How do I fix high p95 latency?

Match the fix to the cause. Queue contention: autoscale, rate limit, or restore capacity. KV cache pressure: cap context length, use paged attention, evict idle cache. Large prompts: cap or route long prompts separately. Batch size too large: cap at latency target. Resource saturation: add capacity or optimize with quantization and model routing. Fixing the wrong cause wastes resources — adding GPUs to a queue problem increases capacity without removing the bottleneck.

Why does p95 latency spike when traffic increases?

Usually queue contention or batch size growth. More traffic means more concurrent requests, which means either a longer queue (requests waiting) or larger batches (sharing GPU time with more peers). Both raise tail latency. The fix is either more capacity to absorb the increased traffic, or rate limiting and batch capping to bound the latency at the higher load.

How do large prompts affect p95 latency?

Large prompts increase time-to-first-token (TTFT) because the prefill phase — processing the prompt and building the KV cache — is compute-heavy and scales with prompt length. If your workload mixes short and long prompts, the long ones create a latency tail. The fix is capping prompt length, routing long prompts to a dedicated model, or optimizing prefill.

Summary

High p95 latency in LLM serving comes from five causes — queue contention, KV cache pressure, large prompts, batch size effects, and resource saturation — each with a distinct diagnostic signature and a specific fix. The diagnostic step is the one that prevents wasted fixes: correlate p95 latency with queue depth, KV cache occupancy, prompt lengths, batch size, and utilization to identify the active cause, then apply the fix that matches it. P95 latency is the metric users experience, the gap between median and p95 is the early warning signal, and the diagnosis that localizes the cause is what turns a latency problem into a solved one. For the full serving monitoring framework, see token generation latency monitoring.