Token Generation Latency Monitoring: Signals That Catch Inference Drift

Token generation latency monitoring is the practice of tracking the time-related signals of LLM serving — time to first token, inter-token latency, queue time, and their tail percentiles — so teams catch inference drift and saturation before users perceive slow responses. The metrics that matter are not averages but tails, because users notice the slow requests, not the typical ones.

For any team serving LLMs in production, latency is the user experience. A model that produces correct answers but responds slowly feels broken, and slow responses erode trust faster than occasional errors. Yet latency is also the metric that drifts silently: as traffic grows, batching changes, or context lengths creep up, latency degrades gradually until it crosses a threshold users notice. Monitoring is how teams detect that drift early rather than after complaints.

This guide covers the latency signals to track, why tails matter more than averages, the patterns that indicate specific problems, and how to set alerts and SLOs that catch issues before users do. It treats latency monitoring as an operational discipline, because that is what keeps LLM serving reliable.

Why Latency Monitoring Is Different for LLM Serving

LLM serving latency is not a single number; it is several. A request has a time to first token (TTFT), which is the delay before the first token appears, and an inter-token latency (ITL) or time per output token (TPOT), which is the pace of subsequent generation. Users perceive both: TTFT determines whether the response feels responsive, and ITL determines whether streaming feels smooth. A serving system can have good TTFT and poor ITL, or vice versa, so both must be monitored separately.

Unlike traditional request-response latency, LLM latency scales with output length and is sensitive to batching and concurrency. A request that generates 500 tokens takes longer than one that generates 50, and a system under high concurrency batches more requests, which affects both TTFT and ITL. Monitoring must account for these variables, or the metrics are not comparable across requests or over time.

The Core Latency Signals

Four signals form the core of token generation latency monitoring. Each captures a different part of the user experience, and together they localize where latency problems originate.

The first is time to first token (TTFT), the delay from request arrival to the first generated token. TTFT captures prompt processing, scheduling, and queueing; a rising TTFT usually signals queue buildup or prompt-processing bottleneck. The second is inter-token latency (ITL), the average time between consecutive output tokens during generation. ITL captures the generation pace; a rising ITL signals compute or memory pressure during decoding. The third is end-to-end request latency, the total time from request to completion, which matters for non-streaming use cases. The fourth is queue time, the time a request waits before any processing begins, which isolates saturation from compute speed.

Why Tails Matter More Than Averages

Track percentiles, not averages. Averages hide the slow requests that users actually experience. A system with a 200ms average latency but a 3-second p99 has a serious problem that the average conceals, because one in a hundred users hits that 3-second latency and perceives a broken service. The p95 and p99 percentiles are the metrics that reflect real user experience, because they capture the tail where dissatisfaction lives.

The gap between average and tail is itself a signal. A small gap means latency is consistent; a large gap means some requests are much slower, which usually indicates contention, queueing, or workload-specific slowdowns. Monitor the gap and investigate when it grows, because a widening average-to-tail gap is often the first sign of saturation before the tail itself crosses an alert threshold.

Core latency signals

Patterns That Indicate Specific Problems

Different latency patterns point to different root causes. Learning to read the patterns speeds diagnosis dramatically. A rising TTFT with stable ITL usually means requests are queuing before processing, which points to insufficient serving capacity or a scheduling bottleneck. A stable TTFT with rising ITL usually means generation itself is slowing, which points to compute or memory pressure during decoding, often from larger batches or KV cache pressure.

A rising tail (p95 or p99) with stable median usually means some requests are hitting a contention point others avoid, such as long-context requests, large batches, or requests that trigger memory pressure. Latency that degrades over hours and recovers after a restart often indicates a resource leak, such as KV cache or memory fragmentation building up until the serving process is recycled. Correlating latency patterns with GPU utilization, batch size, and queue depth localizes the cause much faster than latency alone.

Correlating Latency with System Metrics

Latency metrics in isolation tell you something is slow; correlation with system metrics tells you why. Always pair latency signals with the underlying resource metrics. GPU utilization shows whether the GPUs are saturated; if latency rises while utilization is near peak, the system is compute-bound and needs more capacity or better scheduling. Queue depth shows whether requests are waiting; if queue depth rises while utilization is moderate, scheduling or batching may be the bottleneck rather than raw compute.

KV cache pressure and memory metrics explain ITL drift. If ITL rises while KV cache occupancy is high, the system is spending time on memory management rather than generation, which points to context length, concurrency, or cache configuration. Batch size metrics explain TTFT and throughput tradeoffs. Correlating these turns latency monitoring from a "something is slow" signal into a "here is the bottleneck" diagnosis. Managed AI infrastructure with integrated observability surfaces these correlations so teams diagnose faster.

Alerting on the Right Signals

Set alerts on tails, not averages, and on rates of change, not just thresholds. A p95 or p99 latency alert catches the user-impacting slowdowns that an average alert misses. A rate-of-change alert (latency rising faster than usual over a window) catches drift before it crosses an absolute threshold, which is often the difference between fixing a problem early and firefighting after complaints.

Pair latency alerts with the correlated system metrics. Alerting on p99 latency plus queue depth plus GPU utilization gives context that a latency-only alert cannot, so the on-call engineer knows whether to add capacity, tune scheduling, or investigate a leak. Alerts without context produce noise and fatigue; alerts with correlation produce action. Design the alerting around the diagnosis path, not just the threshold.

Setting Latency SLOs for LLM Serving

A service-level objective turns latency monitoring into a commitment. Define SLOs in terms of percentiles and windows that match user perception: for example, p95 TTFT under a target threshold and p99 end-to-end latency under another, measured over a rolling window. The SLO should reflect what users actually experience, which means percentiles and tails, not averages.

Pair the SLO with an error budget that defines how much violation is tolerable before action is required. An error budget prevents overreaction to brief spikes while ensuring sustained degradation triggers capacity or scheduling changes. Revisit the SLO as the workload evolves; an SLO set at launch may be wrong six months later as traffic patterns, context lengths, and concurrency change. Orchestration platforms with observability help track SLO adherence and error budget burn so teams act before the SLO is breached.

FAQ

What is the difference between time to first token and inter-token latency?

Time to first token (TTFT) is the delay from request arrival to the first generated token, capturing prompt processing, scheduling, and queueing. Inter-token latency (ITL) is the time between consecutive output tokens during generation, capturing the decoding pace. Users perceive both: TTFT determines whether the response feels responsive, and ITL determines whether streaming feels smooth. They can move independently, so monitor both separately.

Why monitor p95 and p99 latency instead of average?

Because users experience the tail, not the average. A system with a low average but a high p99 has slow requests that one in a hundred users hit and perceive as broken. Averages hide these slow requests; percentiles expose them. The gap between average and tail is itself a signal of contention or saturation, so tracking tails catches problems that averages conceal.

What causes latency to drift over time in LLM serving?

Common causes are traffic growth that increases concurrency and batch sizes, context length creep that raises KV cache pressure, scheduling or batching changes that affect TTFT, and resource leaks (such as KV cache or memory fragmentation) that build up until a restart. Correlating latency drift with GPU utilization, queue depth, and memory metrics localizes the cause much faster than latency alone.

How do I set latency SLOs for LLM serving?

Define SLOs in percentiles and windows that match user perception, such as p95 TTFT under a target and p99 end-to-end latency under another, measured over a rolling window. Pair the SLO with an error budget that defines tolerable violation before action is required. Revisit the SLO as the workload evolves, because an SLO set at launch may be wrong as traffic, context, and concurrency change.

How does batching affect token latency monitoring?

Batching changes both TTFT and ITL. Larger batches improve throughput but can raise TTFT (requests wait for the batch to form) and affect ITL (more concurrent decoding). Because batching is dynamic, latency metrics must be interpreted alongside batch size, or comparisons across time or load are misleading. A latency rise that coincides with larger batches may be a throughput tradeoff rather than a degradation, which is why correlation matters.

Summary

Token generation latency monitoring tracks four signals — time to first token, inter-token latency, end-to-end latency, and queue time — with a focus on tails (p95 and p99) rather than averages, because users experience the slow requests. Different latency patterns point to different root causes: rising TTFT signals queueing, rising ITL signals compute or memory pressure, and a widening average-to-tail gap signals contention. Correlate latency with GPU utilization, queue depth, and KV cache metrics to turn "something is slow" into a diagnosis. Set SLOs in percentiles with error budgets, and alert on tails and rates of change with correlated context. Teams that monitor this way catch inference drift and saturation before users do.

For teams that want latency monitoring integrated with capacity and scheduling visibility, managed AI infrastructure with observability surfaces the correlations that speed diagnosis.