Split Prefill and Decode Only When the KV Transfer Is Cheap

Prefill and decode want different things from a GPU. Prefill chews through a prompt and cares about time to first token. Decode emits one token at a time and cares about the gap between tokens. Running both on one device means a long prompt stalls everyone else's decode. Running them on two devices means the KV cache for that request has to move. Papers since Splitwise (Patel et al., 2024) showed the split winning when the first cost dominates. They also imply the unfashionable result: when the move is expensive, leave the phases together. Most architecture slides skip that half.

Two Placements, Not Two Definitions

You are not choosing a vocabulary. You are choosing which cost to pay.

Colocation is the right default for short prompts and modest concurrency, because there is nothing to save and there is a hop to add. A split is the right experiment when prompts are long, decode has a tight per-token budget, and you can name the path the KV will take. Neither placement is "more modern." The paper's speedup is the paper's, on the paper's hardware. Yours is the inequality in the next section.

Dedicated hardware changes a term people smuggle into this table. On a multi-tenant GPU the interference column includes somebody else's batch, not only your own prefill. OneSource Cloud removes that term by giving each pool single-tenant devices: your prefill nodes are not decoding a neighbor's traffic, and your decode nodes are not absorbing a neighbor's prefill. What remains is interference among your own requests. That is the interference a split can still be asked to remove. The host does not remove it for you by existing.

The Transfer Is a First-Class Term

Write it down before you draw the second pool.

split wins when
  time a long prefill stalls a decode on one device
  > time to move that request's KV to the decode pool

The right-hand side is KV size divided by the path you really have, plus whatever the engine spends packing the transfer. KV size follows layers, heads, precision, and context. The path follows whether the pools share a rack, a RoCE fabric, or a VPC detour through a CPU. A design that quotes the left-hand side from a blog and leaves the right-hand side as "RDMA, so it's fine" has not done the comparison.

If you do split, put both pools on a fabric that was built for that hop. OneSource Cloud can place a prefill pool and a decode pool on the same single-tenant RoCE v2 network, so the handoff stays on the dedicated fabric instead of crossing a shared gateway. That is a way to shrink the right-hand side. It is not a reason to skip measuring it. A slow handoff on a fast fabric — small messages, a stage buffer that serializes, a pool scheduled across too many racks — still loses to colocation.

What moves, precisely

The KV for the request, not the weights. Weights are already on the decode pool or you have a different problem. Size the KV from the model you serve and the context you actually accept. Then divide. If you cannot name the number, you cannot defend the split.

Revisit the Split When the Prompt Mix Moves

The inequality is not a migration you finish. Median prompt length drops when a product stops stuffing the context window, and the left-hand side shrinks until colocation is cheaper again. Decode slack shrinks when you tighten the per-token SLO, and the same transfer you tolerated starts missing it. Review on the traffic you have this quarter.

• Collapse the pools when the transfer shows up in time-to-first-token traces more often than prefill interference does.
• Keep the split when long prompts are the common case and the hop stays inside the budget you wrote down.
• Do not keep the split because the diagram is already in production. A diagram is not a measurement.

Questions that stop a premature split

What exactly moves between the pools?

The KV state of that request. Not the weights, not the batch, not a copy of the model. Its size is a function of layers, KV heads, bytes per element, and context length. Divide that by the measured path between the pools, and you have the term the split has to beat. If the engine adds a staging copy, include it; the path's line rate is not the transfer time.

Does OneSource Cloud remove the reason to disaggregate prefill and decode?

No. Dedicated GPUs remove the neighbor's batch from your device. They do not remove the stall your own long prefill causes your own decode. Split the phases only when moving KV across the RoCE path costs less than that stall. OneSource Cloud can make the path dedicated. It does not make the inequality true.