Prefill vs Decode Compute Costs for LLM Inference

Quick Verdict: Prefill is the prompt-processing phase that builds the KV cache from the full input; decode is the token-by-token generation phase that is usually limited by memory bandwidth. Treating “GPU hours” as one number hides the split. Long retrieved contexts burn prefill. Long completions burn decode.

Size the cluster by which phase owns wall-clock time. A 32k RAG prompt with a short answer is a prefill bill. A short system prompt with a long draft is a decode bill. Those shapes need different batching, cache policy, and sometimes different GPU pools. Measure both before you buy more accelerators or change quantization.

Prefill vs decode cost table

Published tokens-per-second figures often mix the two phases. Use them as direction, then split your own traces. A blog that quotes one average hides the phase that actually queues users.

When prefill dominates the inference bill

Prefill cost shows up when the model must attend over a large prompt before it emits the first token. Retrieval-augmented generation is the common enterprise case: each extra chunk is extra prefill, even if the user question is one sentence. Long system policies, few-shot catalogs, and conversation history have the same shape.

The operational failure is familiar. Platform teams watch average GPU utilization, see a busy card, and buy more of the same SKU. Utilization does not say whether the card is multiplying a 20k prompt or streaming a 20-token reply. If TTFT rises with prompt length while decode stays flat, you have a prefill problem. Adding decode-oriented tricks, such as speculative decoding, will not fix the queue at the door.

Capacity planning should treat prompt-length mix as a first-class input. Segment traffic by prompt tokens, not only by request count. A small share of long-context jobs can consume most of the prefill budget and starve short interactive queries on the same replica.

When decode dominates GPU memory and time

Decode is expensive when many sessions stay alive and each keeps a growing KV cache. The model may look “idle” on FLOPs while HBM is full of cache pages. That is why a serving stack can accept a new request on paper and still fail tail latency: the memory is already rented by live conversations.

Decode-heavy fleets care about cache eviction, prefix reuse, and how many concurrent sequences fit. Continuous batching helps when arrivals are bursty and sequences finish at different times. It does not create memory. If decode owns the bill, quantization, cache-aware scheduling, and a hard cap on max output tokens move cost more than another general-purpose training GPU.

Private AI infrastructure helps when you need a dedicated serving pool so a long-context research job cannot steal decode slots from a production chat lane. The isolation is operational, not a claim that one phase is cheaper in the abstract.

How to measure prefill and decode without a fake break-even

Do not hunt for a universal token price where prefill “equals” decode. The crossover depends on model, precision, batch, and prefix-cache hit rate. Run a short lab instead.

1. Fix the model and serving stack, then vary prompt length with a short max-new-tokens cap so prefill is visible in TTFT.
2. Fix a short prompt and vary output length so decode and cache growth show up in inter-token time.
3. Replay a production mix, not a single synthetic prompt, and record p95 for both phases.
4. Repeat with and without prefix cache to see how much “saved prefill” you actually get.
5. Only then change SKU, tensor parallelism, or quantization, and keep the same traces.

If you also need day-2 ownership of those traces, managed AI infrastructure is the layer that keeps metrics, on-call, and capacity reviews from living in a notebook. OnePlus, OneSource Cloud’s AI orchestration platform, is useful when multiple teams share GPUs and you must pin serving pools so a training sweep cannot collapse TTFT. For storage-heavy RAG corpora that inflate prompts, pair the serving plan with AI storage architecture so retrieval does not silently lengthen every prefill.

FAQ

What is prefill versus decode in LLM inference?

Prefill processes the entire prompt and writes the key-value cache. Decode then generates tokens one step at a time using that cache. Prefill usually shows up as time to first token. Decode shows up as inter-token latency. Cost follows the phase that occupies the GPU for your real traffic mix, not a single average tokens-per-second number.

Does a longer context window always raise inference cost?

It raises prefill work and KV-cache memory. Whether the monthly bill moves depends on how often those long prompts actually run, whether prefix cache hits, and how long the model is allowed to generate. A rarely used 128k path is a capacity reservation. A default 128k path on every call is a standing prefill tax.

How should we allocate GPUs between RAG and chat?

Split pools when the prompt-length distributions disagree. RAG and policy-heavy prompts need prefill headroom and retrieval locality. Chat and agent loops need decode concurrency and cache policy. Sharing one replica is fine for a lab. It is a poor production default once p95 TTFT and p95 inter-token latency start trading off against each other.

Can quantization cut prefill and decode equally?

No. Weight quantization can help both, but decode is more sensitive to memory bandwidth and cache size, while prefill is more sensitive to compute throughput on the prompt. Measure each phase after you quantize. A method that looks good on a short-prompt chat bench can still lose on long retrieved contexts.

When does private inference infrastructure change this math?

Dedicated hardware does not erase prefill or decode physics. It changes whether a noisy neighbor or a shared quota can steal the phase you already sized. OneSource Cloud’s U.S. dedicated environments are relevant when you must keep a serving pool stable and resident, then tune phases on hardware you do not share.

Summary

Prefill vs decode is a cost-shape decision, not a slogan. Split traces, segment prompt and output lengths, and buy capacity for the phase that queues users. If you need a dedicated serving environment to keep that plan intact, review OneSource Cloud private AI infrastructure before you treat another GPU SKU as the fix.