The KV Cache in LLM Inference Explained

The KV cache is the attention state an LLM stores during generation so it does not have to recompute the prompt's representations for every new token — and it is both the reason autoregressive inference is fast and the reason it consumes so much GPU memory. Understanding the KV cache is the key to understanding why LLM serving behaves the way it does.

For anyone building or operating LLM serving, the KV cache is not an implementation detail; it is the dominant factor in serving memory use, the reason long contexts are expensive, and the lever behind most inference optimizations. A team that does not understand the KV cache cannot reason about serving cost, latency, or concurrency, because all three trace back to it. This concept is foundational for every later serving decision.

This guide explains what the KV cache is, how it speeds inference, why it drives memory use, how it scales, and what optimizations manage it. It treats the KV cache as the central object in LLM serving, because that is what it is in practice.

What the KV Cache Actually Is

The KV cache stores the key and value tensors that an LLM's attention mechanism computes for each token it has seen, so that generating the next token does not require recomputing those tensors from scratch. When an LLM generates text autoregressively, each new token must attend to all previous tokens. Without a cache, the model would recompute the key and value representations for the entire prompt and all generated tokens at every single step — work that grows quadratically with sequence length and makes generation impractically slow. The KV cache stores those representations once, during the prefill phase, and reuses them for every subsequent token.

The cache is called "KV" because attention computes two tensors per token — keys and values — and both are cached. (Queries are recomputed each step because only the current token's query is needed.) The cache holds, for every layer and every attention head, the key and value for every token processed so far. This is what makes generation efficient: instead of redoing the attention math for the whole sequence at every step, the model reads the cached keys and values and only computes the new token's contribution.

How the KV Cache Speeds Inference

The KV cache speeds inference by turning a quadratic-cost problem into a linear one. Without the cache, generating the Nth token requires attending to all N-1 previous tokens, which means recomputing their representations — work that grows with sequence length and makes each successive token slower than the last. With the cache, those representations are already stored, so generating the Nth token only requires computing the new token's query, keys, and values and attending against the cached keys and values. The per-token cost stays roughly constant regardless of how long the sequence has grown.

This is why LLM serving has two distinct phases with different cost profiles. Prefill, which processes the prompt and builds the initial KV cache, is compute-heavy and its cost scales with prompt length. Decode, which generates tokens one at a time using the cache, is memory-bandwidth-bound and its per-token cost stays roughly flat. The KV cache is what makes decode fast enough to be usable; without it, every generated token would get slower as the sequence grew, and generation would grind toward impracticality.

The two inference phases and the KV cache

Why the KV Cache Drives Memory Use

The KV cache's speed comes at a memory cost, and that cost is substantial. The cache stores keys and values for every layer, every attention head, every token processed — so its size scales with model size (more layers and heads), context length (more tokens cached), and concurrency (each active request has its own cache). For long-context workloads or high-concurrency serving, the KV cache can exceed the memory used by the model weights themselves, which is why it often dominates serving memory rather than being a minor overhead.

This has practical consequences. The KV cache sets how much GPU memory a serving deployment needs, which sets how many concurrent requests a fixed GPU pool can serve. A larger cache (from longer contexts or more concurrency) means fewer concurrent requests fit, which means lower throughput or more GPUs. This is why context length and concurrency are so consequential for serving cost: they do not just affect latency, they affect memory, which affects how many requests the hardware can serve at once. Infrastructure sized for LLM serving must account for the KV cache, not just the model weights.

How the KV Cache Scales

The KV cache grows with three factors, and understanding the scaling is essential for planning serving capacity. First, model size: more layers and attention heads mean more cached tensors per token. Second, sequence length: the cache holds every token processed, so longer prompts and longer generations produce larger caches. Third, concurrency: each active request maintains its own cache, so more simultaneous requests multiply the memory. The interaction of these factors means a workload with long contexts at high concurrency can require dramatically more memory than the model weights alone would suggest.

This scaling is why long-context LLMs are expensive to serve, and why concurrency is capped by memory rather than compute in many serving setups. A model that fits comfortably in GPU memory for a single short request may not fit dozens of long-context requests simultaneously, because the aggregate KV cache exceeds available memory. Planning serving capacity means planning for the KV cache at the intended context length and concurrency, not just for the model in isolation.

How Serving Systems Manage the KV Cache

Because the KV cache dominates memory, serving systems have evolved techniques to manage it. Three are widely used. First, capping maximum context length prevents runaway cache growth, trading capability for predictable memory use. Second, evicting or compressing the cache for idle or completed requests reclaims memory, though eviction can hurt quality if a request resumes and its cached state is gone. Third, paged or chunked attention allocates cache memory dynamically rather than reserving worst-case space per request, which dramatically improves memory efficiency and the number of concurrent requests a GPU can serve.

These techniques are not just optimizations; they are often the difference between a serving deployment that is economical and one that is not. A naive serving setup that reserves worst-case cache per request wastes memory and serves few concurrent requests. A setup with paged attention and sensible eviction policies serves many more requests on the same hardware, which directly lowers cost per token. For teams optimizing serving, the KV cache management policy is one of the highest-leverage decisions.

What the KV Cache Means for Serving Cost and Latency

The KV cache ties together the three metrics that define serving economics. It drives memory use, which sets how many concurrent requests fit and thus throughput. It drives latency, because decode must read the cache each step, so a larger cache (longer context) means slower per-token generation as more data moves through memory. And it drives cost, because the memory it consumes must be provisioned whether or not it is fully used. Managing the KV cache well lowers all three; managing it poorly inflates all three.

This is why the KV cache is the central object in LLM serving optimizations like batching, quantization, and paged attention — each of these interacts with the cache. Batching reuses the model weights across requests but each request still has its own cache, so batching raises throughput without proportionally raising cache memory. Quantization can shrink the cache along with the weights. Paged attention makes cache allocation efficient. Understanding the cache is the prerequisite to understanding why these optimizations work and where they trade off. Managed AI infrastructure with serving runtimes that implement these techniques extracts far more from the same GPU memory than a naive setup.

FAQ

What is the KV cache in LLM inference?

The KV cache stores the key and value tensors an LLM's attention mechanism computes for each token it has seen, so generating the next token does not require recomputing them from scratch. It holds, for every layer and attention head, the keys and values for every token processed. The cache is built during prefill and reused and extended during decode, which is what makes autoregressive generation fast enough to be practical.

How does the KV cache speed up LLM inference?

Without the cache, generating each new token would require recomputing attention representations for all previous tokens, with cost growing as the sequence gets longer. The cache stores those representations once, so each new token only computes its own contribution and reads the cached keys and values. This turns a quadratic-cost problem into a roughly linear one, keeping per-token generation cost flat regardless of how long the sequence has grown.

Why does the KV cache use so much memory?

Because it stores keys and values for every layer, every attention head, and every token processed, and because each active request has its own cache. Its size scales with model size (more layers and heads), context length (more tokens cached), and concurrency (more simultaneous requests). For long-context or high-concurrency workloads, the aggregate KV cache can exceed the memory used by the model weights themselves.

Does the KV cache make long context expensive?

Yes. The cache grows with sequence length, so longer prompts and longer generations produce larger caches that consume more memory and slow decode (more data to read each step). This is why long-context LLMs are expensive to serve: they need more memory per request, which means fewer concurrent requests fit on a GPU, which means lower throughput or more hardware. Planning serving capacity means planning for the cache at the intended context length.

What is paged attention and how does it help the KV cache?

Paged attention allocates KV cache memory dynamically in small blocks rather than reserving worst-case space for each request upfront. This dramatically improves memory efficiency, because requests that use short contexts do not waste memory reserved for the maximum, and more concurrent requests fit on the same GPU. It is one of the most effective techniques for raising serving throughput without adding hardware, by making the cache's memory footprint match actual usage rather than worst-case assumptions.

Summary

The KV cache is the attention state an LLM stores during generation so it does not recompute prior tokens' representations at every step. It is what makes autoregressive inference fast, by turning a quadratic-cost problem into a roughly linear one, and it is what makes inference memory-hungry, because it stores keys and values for every layer, head, and token across every active request. Its size scales with model size, context length, and concurrency, which is why long contexts and high concurrency are expensive to serve. Serving systems manage the cache by capping context, evicting idle state, and using paged attention to allocate memory dynamically. The KV cache ties together serving memory, latency, and cost, which is why it is the central object in nearly every LLM serving optimization.

For teams that want serving runtimes that manage the KV cache efficiently, managed AI infrastructure with modern serving techniques extracts far more throughput from the same GPU memory.