When Reloading a KV Cache Loses to Just Recomputing It

Offloading a KV cache to CPU or NVMe is sold as a way to stop recomputing long prefixes. That sale skips the comparison that decides it. Reloading has a transfer time. Recomputing has a prefill time. Whichever is smaller is the one you want, and on a fast GPU with ordinary prompt lengths the prefill often wins. A 2026 vLLM study (Kanichai, De Matteis, Trivedi) measured exactly that split: external KV reuse helps long prefixes and slower GPUs, and on an H100 replay of a production trace the average request sat below the break-even. GPU memory alone was enough.

The Prefix Length Where Disk Loses

Treat reload as an admission decision, not as a tier you enable for the fleet. The two times you are comparing:

The paper's useful result is not a product ranking. It is that the average request on a strong GPU never reached the length where disk paid. Teams who turn NVMe offload on because a blog showed an 80k-token win are optimizing a prefix their traffic does not have. Measure your own prefix-length distribution first. If the median is a system prompt plus a page of context, you are in the recompute region, and the SSD is a more expensive way to get a worse time to first token.

There is a second, quieter cost. A shared offload tier lets one tenant's cold sessions occupy the staging path another tenant needs for a hot prefix. Dedicated GPUs with their own storage path avoid that cross-talk. OneSource Cloud attaches NVMe-oF to the same single-tenant nodes as the GPUs, with a GPUDirect path so a reload does not take a needless CPU copy. That shortens the reload column in the table. It does not move a short prefix out of the recompute column.

Bandwidth Is the Wrong First Variable

Quotes for this design love a peak NVMe figure. The characterization found the result moving with three things that are not that figure.

• Transfer granularity. A cache moved in awkward chunks spends the hop on overhead. Bytes per second of the device never enter the picture.
• Staging memory. An unbounded stage evicts something else you needed on the GPU. A bounded stage serializes the reloads you hoped would overlap.
• When the read starts. A read that begins while the request is still queued overlaps the wait. A read that begins after the GPU is free adds itself to time to first token. The same paper attributed a large part of its gain over a baseline connector to that overlap, not to a faster disk.

Older offload stacks have been measured using a fraction of the SSD's peak, because the software schedule, not the NAND, was the limit. If a vendor's slide and your achieved refill bandwidth disagree by several times, believe the refill.

Assumptions a number must state

Prefix length, GPU class, whether CPU and disk were both enabled, and whether the read overlapped the queue. A number missing any of those is not comparable to yours. Do not import an 80k-token result into a 4k-token product.

Pick the Tier by Idle Time, Not by Fashion

Once a prefix is long enough to be worth keeping, where it sits should follow how soon it will be read again. A workable heuristic, used as a heuristic and not as a law:

Give each tenant its own TTL. A global "move to disk after ten minutes" policy is how a quiet neighbor's sessions push a busy tenant's prefixes out of CPU. On a single-tenant node that failure mode is inside your own scheduler, which is a better place for it: you can see the queue, and nobody else's idle chat is on the drive.

Revisit the policy when the prefix mix shifts. A product that adds a long retrieved context can cross the break-even in a release. A product that shortens its system prompt can fall back under it. The tier is a consequence of the distribution, not a commitment to a storage architecture.

Questions the break-even raises

Should system prompts live on NVMe?

Only when they are long and reused enough that a reload beats running the prefill again. A short shared system prompt is a GPU or CPU resident. Putting it on NVMe adds a hop to every new session in exchange for almost no saved FLOPs. Long, repeated retrieval prefixes are the ones that belong on the cold tier.

Does OneSource Cloud's storage make every KV reload faster than recomputing?

No. OneSource Cloud's NVMe-oF tier and GPUDirect path shorten the refill by keeping it on the tenant's own fabric and off an extra CPU copy. A short prefix on a fast GPU is still cheaper to recompute than to reload. The storage changes the cost of one column. Your prefix lengths decide which column wins.