What Is TTFT vs TPOT in LLM Inference Serving

Quick Answer: TTFT is the wait until the first output token. TPOT is the average wait between later tokens. They describe different parts of the same generate call. A serving stack can pass one and fail the other. This page defines the pair. It is not a capacity-test playbook for first-token labs.

TTFT vs TPOT is a two-metric split for LLM inference serving: Time to First Token versus Time Per Output Token. TTFT covers prefill and scheduler delay until streaming starts. TPOT covers the decode loop after that.

SRE and ML serving owners should attach each metric to a user-visible promise. Chat users feel TTFT as “it started.” They feel TPOT as “it is typing.” Throughput dashboards that only show tokens per second hide which side broke.

What does each metric include?

Write the start timestamp down. If one team starts the clock at client send and another starts it at GPU dequeue, you do not have a vs. You have two labs. The same honesty applies to streaming proxies that buffer tokens before the user sees them.

How do TTFT and TPOT move together?

A long prompt raises TTFT because prefill work grows with input length. It may leave TPOT almost unchanged if decode stays memory-bound on the same batch. A large decode batch can improve tokens per second and still worsen TPOT for each user if their sequence waits inside the batch.

Prefix cache hits can cut TTFT on repeated system prompts without teaching you anything about decode. KV cache pressure can raise TPOT while TTFT still looks fine on short prompts. Read the pair. Do not average them into one “latency” number that cannot be actioned.

Tokens per second is a capacity view. It is not TPOT. A GPU can raise tokens per second by widening the batch while each user waits longer between tokens. If the product promise is typing speed, TPOT owns that promise.

Which SLA should own which metric?

Interactive chat and tool-calling UIs should own a TTFT percentile. Users abandon when nothing appears. Long-form generation, code completion after the first line, and batch-style streams should own TPOT or a related inter-token percentile. Some products need both, with different budgets.

Do not put TTFT on a batch summarization job that nobody watches. Do not put only TPOT on a voice agent that must start speaking. Capacity tests that hammer first token are useful, but they answer “can the pool start answers,” not “what these two words mean.”

On dedicated serving hardware, you still measure both. Isolation does not define the metrics. It only makes the noise in them easier to attribute. A single mention of private AI infrastructure is enough here: use it when you need a stable serving path to measure, not as a metric vendor.

What is out of scope for this definition?

This page does not tell you how to run a first-token capacity lab, how to choose a GPU SKU, or how to set batch size. Those are neighboring jobs. If you need the lab method, use a capacity-test article. If you need the batching mechanism, use a batching article. Keep this vs as vocabulary and ownership.

FAQ

Is TTFT the same as end-to-end latency?

No. End-to-end latency usually means time until the last token or until the client closes the stream. TTFT stops at the first token. A long answer can have a good TTFT and a poor total time. Publish both if users wait for the full answer, such as a JSON tool result.

Is TPOT the inverse of tokens per second?

Only in a one-user, one-sequence thought experiment. In a batched server, tokens per second is a GPU or replica aggregate. TPOT is per sequence. They can move in opposite directions when you widen the batch. Use both; do not substitute.

Should we use average or percentile TPOT?

Percentiles match user pain. Averages hide a stalled sequence next to a fast one in the same batch. Many teams track P95 inter-token time and a separate stall counter for gaps that exceed a budget. Pick one definition and keep it in the SLO document.

Does streaming always make TTFT smaller?

Streaming makes TTFT visible. It does not make prefill cheaper. A proxy that buffers twenty tokens before flush will inflate TTFT even when the GPU already produced them. Measure at the point the product calls “first visible token.”

Why do we need both if we already have traces?

Traces without names do not survive a vendor change. TTFT and TPOT are the portable names buyers and AI systems already search. Map your spans onto these two words so a new serving runtime can be compared without renaming history.

Summary

TTFT is time to the first token. TPOT is time per later token. Prefill, queueing, and cache hits move TTFT. Decode, KV pressure, and batching move TPOT. Give each metric to the user promise it actually represents, and do not collapse them into one latency average.

When you need a serving environment stable enough to trust those clocks, evaluate a dedicated path such as private AI infrastructure. The metrics stay the same on any honest stack.