How Paged Attention Manages Inference KV Cache

Quick Answer: Paged attention manages the inference KV cache by allocating it in fixed-size blocks instead of one contiguous reservation per sequence. The serving process maps a sequence’s logical token positions onto those blocks, which cuts waste from over-reserve and makes sharing or freeing a prefix cheaper. It does not change the attention formula and it does not add GPU memory that was not there.

Paged attention is a KV-cache memory manager for LLM inference that stores keys and values in blocks and translates logical token indexes to physical pages. The name is an analogy to virtual memory, not a claim that attention is offloaded to disk by default.

Serving engineers should use this page to understand allocation behavior. It is not a long-context capacity-planning guide and not a definition of what the KV cache is.

What problem does paging solve?

The manager’s job is to accept, grow, share, and evict sequences without giving every request the worst-case slab. If your traffic is one long job at a time, paging helps less and a simple reservation may be easier to reason about.

How are blocks mapped during a request?

Prefill writes KV for the prompt into as many blocks as the token count needs. Decode appends a slot, allocating a new block when the current one is full. The attention kernel reads through a block table, so logical position i is not required to sit at physical address i times hidden size.

When a sequence finishes, its private blocks return to the free list. Shared prefix blocks return only when the last reference drops. That is why prefix caching and paged attention often appear in the same serving stack: one decides what may be reused, the other decides how the bytes are packed.

Eviction is a policy on top of paging, not paging itself. A server can page in memory and still evict a sequence’s blocks when the pool is full. An OOM means the free list could not satisfy the next block, not that attention math failed.

What paging does not do

It does not make a 128k context cheap. Long contexts still store more KV. Paging reduces internal waste. It does not erase the bytes. Capacity planning for long context remains a separate measurement of tokens times layers times heads times width.

It does not guarantee isolation between tenants. Block tables are process state. If two customers share a runtime, you still need a tenancy design. Paging is not MIG. It is not a BAA.

It does not require a particular GPU brand story. It is software layout on whatever device memory you already have. A dedicated serving pool makes the free-list behavior yours to measure. One mention of private AI infrastructure belongs here as that stable pool, not as a paged-attention feature.

FAQ

Is paged attention the same as OS swap?

No. The usual serving design keeps KV in GPU memory and pages inside that pool. Some systems can offload KV to CPU or disk, but that is an extra path with its own latency. Do not hear “paged” and assume the model is streaming from a hard drive.

Does paging change answers?

It should not, if the kernel reads the same logical KV. Answer drift after a serving change is more often tokenizer, template, or sampling. If you suspect a paging bug, compare outputs on a frozen eval with paging on versus a contiguous baseline, and treat mismatches as defects, not as a new style.

Why do we still OOM with paged attention?

Because the live set of blocks can still exceed device memory. Concurrent long sequences, large batches, and unevicted prefixes add up. Paging removes a class of reserved-but-unused waste. It does not create memory. Watch peak live blocks, not only average tokens.

How is this different from prefix caching?

Prefix caching is a reuse policy for a matching lead across requests. Paged attention is the allocator that can make that reuse cheap. You can page without sharing prefixes. You should not share prefixes without an allocator that can point at the same blocks safely.

Do we need this concept to choose a GPU?

Not as a SKU name. You need it to interpret memory dashboards and to avoid over-reserving max length for every request. GPU choice still starts from weights, live KV, and SLO. Paging changes how efficiently you pack the KV term.

Summary

Paged attention manages inference KV cache as blocks plus a translation table. It reduces contiguous over-reserve and enables cheaper prefix sharing. It does not change attention math, does not add memory, and does not isolate tenants. Read OOMs as a free-list miss, not as a reason to abandon the idea.

When you want those free-list measurements on a pool nobody else can steal, evaluate private AI infrastructure as the environment and keep paging as a serving-runtime property.