Prefill vs Decode GPU Capacity Planning for Inference SLAs

Prefill is the GPU work that reads the prompt and fills KV cache, and decode is the GPU work that emits output tokens one step at a time; capacity planning that treats them as one “inference GPU” will miss SLAs. Long prompts saturate prefill. Chatty outputs saturate decode. A single replica count cannot be right for both on every traffic mix.

This article is a planning split, not a kernel cookbook. You still need a serving stack. You also need a GPU pool that can hold the heavier phase without stealing the other. Existing cost write-ups of prefill versus decode are not a substitute for an SLA-sized reservation.

Two phases, two bottlenecks

A farm that looks 40% utilized can still miss TTFT if every request is a long RAG prompt arriving together. Utilization averages hide phase contention. Plan to the percentile and the phase, not to a farm-wide SM chart.

How to size without fake benchmarks

Measure your own prompt-length histogram and output-length histogram on production-like traces. Size prefill for the tail prompt, not the mean. Size decode for concurrent conversations, not for single-stream tokens/sec from a blog. If one pool does both, add headroom for the collision: a prefill batch that stalls decode, or a long decode batch that delays the next prefill.

Disaggregating prefill and decode onto different GPUs is a valid architecture when the mix is extreme. It is also more moving parts. Do not split until you have traces that show one phase dominating. A split on exclusive GPUs is operationally cheaper than a split across noisy public instances whose neighbors change the tail.

Where the pool has to live

SLA inference wants reserved GPUs, not fair-share leftovers from training. Prefill spikes will look like idle decode cards. Do not reclaim those cards into a training job if TTFT is a product promise. That is the same reservation lesson as mixed training versus serving, applied inside the serving stack.

OneSource Cloud’s private AI infrastructure is exclusive capacity you can partition for serving phases. OnePlus, OneSource Cloud’s AI orchestration platform, keeps those partitions away from research burst. Low-latency paths still depend on AI networking when decode is distributed. Managed operations watch TTFT and tokens/sec as different graphs so a “healthy GPU util” does not hide a missed SLA.

FAQ

Why separate prefill and decode in GPU capacity planning?

Because they scale with different traffic shapes. Prompt-heavy RAG hurts prefill and TTFT. Long answers hurt decode and streaming latency. One replica number sized on average tokens will be wrong for the tail of one phase. Plan both, then decide whether they share a GPU pool.

Can one GPU do both prefill and decode for an SLA?

Yes for many mid-size models if you batch carefully and reserve enough replicas. No if the prompt tail and the concurrency tail both want the whole device at once. Use traces, not a vendor demo, to decide. If TTFT and tokens/sec never fail together, you may still be fine on one pool.

Does disaggregated prefill/decode require more GPUs?

It requires more partitions, which often means more cards and more networking. It can still be cheaper than oversizing a combined pool for both tails. It is not a free lunch. Operate the split only when measurement says one phase is starving the other in production-like load.

How does KV cache change the plan?

A larger cache can reduce prefill for repeated prefixes and increase memory pressure on decode. Capacity planning has to include HBM, not only SM count. If cache misses dominate, you are back to prefill-bound. If cache hits dominate, decode and memory become the constraint.

Should training GPUs cover prefill spikes?

Only with an explicit borrow and a reclaim faster than the SLA. Otherwise a fine-tune will own the cards when users arrive. Serving phases belong on reserved exclusive GPUs. Training can wait. Users will not.

Summary

Plan GPU capacity for prefill and decode as two constraints, then reserve serving away from training. Size from your prompt and output tails, not from a mean utilization chart. If exclusive partitions are how you keep the SLA honest, use OneSource Cloud private AI infrastructure and hold those partitions on OnePlus.