When to Disaggregate Prefill and Decode for Inference

Quick Answer: Disaggregate prefill and decode when one GPU pool cannot keep both stages healthy at the same time: long prompts stall token streams, or decode replicas sit idle while a few fat prefills occupy the cards. Stay unified when traffic is short, concurrency is modest, and a second fabric for KV handoff would cost more than it returns.

Prefill/decode disaggregation is an inference-pool design that runs prompt processing and token generation on different GPUs and moves KV state between them. It is an operating decision about two queues, not a new definition of prefill or decode, and not a batching tutorial.

Serving owners should treat this as a split-or-not choice on private GPUs they already control. This page does not re-teach stage math, prefix caches, or paged attention. Those are separate questions.

What problem is disaggregation actually solving?

If the only evidence is a slide about “PD disaggregation,” stay unified. The split adds a control plane, a KV transport, two autoscalers, and two failure domains. That is justified when the queue shapes disagree, not when the vocabulary is fashionable.

When should a private GPU estate split the pools?

Split when your production mix is bimodal. A clerk chatbot with 1–2k-token prompts can share a pool. A document assistant that swallows 50k-token packs cannot share that pool with a high-QPS short decode without a policy. Disaggregation is one policy. Another is “long jobs go to a batch prefill queue on the same cards.” Prefer the simpler policy if it meets the SLO.

Split when you can name two different GPU shapes. Prefill often wants higher memory bandwidth per new token of context. Decode often wants more replicas for concurrency. If both stages want the same SKU in the same count, a split will not save cards. It will only add a network hop.

Split only on exclusive GPUs you schedule yourself. Shared tenancy makes KV handoff and noisy neighbors indistinguishable. Dedicated private AI infrastructure is the right setting to try a split because you can pin prefill nodes and decode nodes without a neighbor reset wiping the experiment.

When should you refuse to disaggregate?

Refuse when the fabric between nodes is the training network you already know is busy, or a commodity Ethernet hop with no measured KV bandwidth. Disaggregation moves large KV blocks. If that copy is slower than finishing prefill on the decode card, you bought a slower unified server.

Refuse when the team cannot operate two SLOs. Prefill has a queue time and a compute time. Decode has inter-token time and a max concurrency. If on-call still has one dashboard named “GPU util,” the split will page the wrong people.

Refuse when you have not yet capped prefill concurrency on the unified server. Many “we must disaggregate” tickets die after a max-prefill-jobs setting and a separate long-context queue. Do that in a week. Build a KV fabric only after it fails.

How do you trial a split without rewriting the estate?

Take one model and one traffic class. Build a prefill pool and a decode pool with explicit KV export and import. Freeze weights and tokenizer hashes. Run the same trace unified and split. Compare user-visible first token, inter-token delay, and GPU seconds per successful request. If GPU seconds rise and the SLO does not improve, roll back.

Count operational objects: two images, two HPAs, one KV bus, one timeout policy for a lost prefill. If you cannot draw that on one page, you are not ready. OnePlus Platform, OneSource Cloud's AI orchestration platform, should show the two pools as different quotas so decode cannot be starved by a prefill backfill.

OneSource Cloud can host both pools on exclusive U.S. GPUs, including Texas / Richardson, and keep east-west traffic on a fabric you can name. If the provider cannot tell you where KV travels, do not disaggregate there. Networking detail belongs on the high-performance AI networking path, not in a serving slogan.

FAQ

Does disaggregated prefill and decode always cost more?

It costs more control plane and usually more network. It saves GPU time only when a unified pool was wasting decode slots on fat prefills, or wasting prefill-capable cards on tiny decodes. Measure GPU seconds per successful request. If that number rises, the split is an expensive rename.

Is this the same as prefill versus decode cost accounting?

No. Cost accounting explains which stage burns the card on a unified server. Disaggregation decides whether those stages own different cards. You can account without splitting. You should not split without accounting.

Do we need a special inference stack to split?

You need a stack that can export KV and schedule the two stages independently. The brand of that stack is secondary. If your current server cannot hand off KV, you do not have a disaggregation project. You have a migration project. Keep that scoped.

Should every long-context model be disaggregated?

No. A low-QPS long-context job can run unified on exclusive GPUs with a concurrency of one or two. Disaggregation pays off when long prefills and short decodes fight in the same batcher all day. Look at the mix, not the context window on the model card.

How does this change capacity reservations?

You reserve two minima instead of one. Prefill can scale with document intake. Decode can scale with concurrent chats. Managed AI infrastructure helps only if on-call has runbooks for KV timeouts as well as for a dead replica. Two pools double the ways to fail.

Summary

Disaggregate when long prefills and live decodes harm each other on the same exclusive GPUs and you already have a KV path you can measure. Stay unified when traffic is short, concurrency is moderate, or the handoff fabric is a guess.

Trial one model, compare GPU seconds and user SLOs, and keep the two pools named. Private U.S. GPU estates make that trial honest. They do not make a split mandatory. Start from OneSource Cloud private AI infrastructure only after a unified cap has failed in production.