What Is LLM Inference? How Trained Models Generate Responses

LLM inference is the phase where a trained large language model generates responses to new prompts, running the model's learned weights forward to produce text one token at a time — and it is the phase that dominates the real-world cost, latency, and infrastructure of putting an LLM into production. Training builds the model; inference is what users actually experience.

For anyone deploying generative AI, understanding inference matters because it is where the operational reality lives. A model that trained successfully can still fail in production if its inference is too slow, too expensive, or too resource-hungry to serve at the required scale. Inference is the bridge between a capable model and a working product, and the challenges of crossing that bridge are distinct from the challenges of training.

This guide explains what LLM inference is, how it works mechanically, how it differs from training, what it costs and why, and what infrastructure it requires. It is the foundation that makes the surrounding topics — cost optimization, latency monitoring, GPU selection — make sense.

What LLM Inference Actually Is

Inference is the use of a trained model to produce outputs. For an LLM, that means taking a prompt (input text), running it through the model's neural network, and generating a response (output text). The model's weights — the parameters learned during training — stay fixed during inference; no learning happens. The model applies what it learned to new inputs it has never seen.

The defining mechanic of LLM inference is autoregressive generation. The model does not produce a full response at once; it generates one token (a piece of a word) at a time, and each new token is predicted based on the prompt plus all the tokens generated so far. This is why an LLM response streams in piece by piece, and it is why generation length directly affects latency: a 500-token response takes roughly ten times longer to generate than a 50-token response, because each token depends on the previous ones.

How LLM Inference Works Mechanically

Each inference request has two phases. The first is prefill, where the model processes the prompt: it reads the input tokens, computes their representations, and builds the attention state (the KV cache) that will be reused during generation. Prefill is compute-heavy because it processes the entire prompt at once, and its cost scales with prompt length. The second is decode, where the model generates output tokens one at a time, each using the KV cache from the prompt and all prior generated tokens. Decode is memory-bandwidth-bound, because each token requires moving weights and KV cache through memory.

This two-phase structure explains several practical behaviors. Time to first token (TTFT) is dominated by prefill, so long prompts mean slow first tokens. Inter-token latency during generation is dominated by decode, which is bounded by memory bandwidth rather than peak compute. The KV cache grows with both prompt length and the number of generated tokens, which is why long conversations and long outputs consume substantial GPU memory. Understanding these mechanics is the key to understanding inference cost and latency.

The two phases of inference

How Inference Differs from Training

Training and inference are often confused, but they are fundamentally different operations with different demands. Training adjusts the model's weights using large datasets and runs both forward and backward passes (to compute gradients), making it compute-bound and data-hungry. Inference uses fixed weights to produce outputs and runs only forward passes, making it memory-bandwidth-bound for autoregressive generation. Training is a batch operation done periodically to produce or update a model; inference is a serving operation done continuously to respond to user requests.

The infrastructure implications differ accordingly. Training needs peak compute and high inter-GPU bandwidth for the collective operations in distributed training, and it tolerates latency because a training run takes hours or days. Inference needs enough GPU memory to hold the model and KV cache, enough bandwidth for fast decode, and low latency because users are waiting. This is why the GPU choices, cluster designs, and optimization techniques for training and inference are different — they optimize for different bottlenecks.

Why LLM Inference Is Expensive

Inference cost comes from three sources. The first is GPU memory: the model weights and KV cache must reside in GPU memory for the duration of serving, which means GPUs are held even when not actively generating. The second is compute: each generated token requires a forward pass, so cost scales with the number of tokens produced. The third is utilization: a serving system that runs at low average utilization pays for GPUs that sit idle, which inflates cost per token.

The expense compounds with scale in ways training does not. Training is a one-time (or occasional) cost; inference is a continuous cost that grows with every user and every request. A model that costs little to train once can cost far more to serve over months of production traffic. This is why inference cost optimization — batching, quantization, model routing, and infrastructure matching — is a major operational discipline for any team serving LLMs at scale.

What Infrastructure LLM Inference Requires

LLM inference requires GPU infrastructure sized for the model's memory footprint and the workload's latency and throughput targets. The model weights must fit in GPU memory (or be split across GPUs with tensor parallelism), and enough memory must remain for the KV cache, which scales with concurrency and context length. The GPU's memory bandwidth determines decode speed, which is why bandwidth, not just peak FLOPS, is the decisive spec for inference throughput.

For models larger than a single GPU can hold, inference uses multi-GPU tensor or pipeline parallelism, which adds inter-GPU communication to every forward pass and makes the interconnect relevant. For production serving, the infrastructure must also handle batching (to keep GPUs saturated across many concurrent requests), scaling (to meet traffic peaks), and monitoring (to catch latency and cost drift). Dedicated GPU infrastructure sized for inference workloads provides the memory, bandwidth, and operational support that production serving requires.

Key Metrics for LLM Inference

Four metrics define inference performance and economics. Throughput, measured in tokens per second, captures how much the system produces. Latency, split into time to first token and inter-token latency, captures how responsive the system feels. Cost per token, computed as GPU cost divided by tokens produced, captures the economics. And utilization, the fraction of GPU capacity actually used, captures efficiency. These four metrics interact: higher batching raises throughput and utilization but can raise latency; optimization techniques like quantization lower cost per token but can affect quality. Production inference management is the discipline of balancing these metrics against the workload's requirements.

Batching: Why It Matters So Much

Batching is the single most important technique for efficient LLM inference. Because decode is memory-bandwidth-bound, a GPU serving one request at a time wastes most of its parallel capacity: the weights and KV cache move through memory for a single token, leaving compute idle. Batching multiple requests together reuses that memory movement across many tokens, dramatically raising throughput and lowering cost per token.

Continuous batching (iteration-level batching) is the modern standard because it admits and evicts requests mid-generation rather than waiting for a batch to complete, which fits the variable-length nature of conversational traffic. The tradeoff is implementation complexity and some tail latency impact, but the throughput gains usually dominate. Teams that skip batching or run with tiny batches pay far more per token than necessary, which is why batching is the first lever in any inference cost optimization effort.

FAQ

What is the difference between LLM training and inference?

Training adjusts the model's weights using large datasets and runs forward and backward passes, making it compute-bound and data-hungry. Inference uses fixed weights to generate outputs and runs only forward passes, making it memory-bandwidth-bound for autoregressive generation. Training is a periodic batch operation to produce or update a model; inference is continuous serving to respond to requests. They optimize for different bottlenecks, so their infrastructure and techniques differ.

Why is LLM inference expensive?

Inference cost comes from GPU memory (weights and KV cache must reside in memory for the duration of serving), compute (each token requires a forward pass, so cost scales with tokens produced), and utilization (idle GPUs inflate cost per token). Unlike training, which is a one-time cost, inference is continuous and grows with every user and request, which is why it often dominates the total cost of running an LLM in production.

What is the KV cache in LLM inference?

The KV cache is the attention state computed during prefill and reused during decode. It stores intermediate representations of the prompt and generated tokens so the model does not recompute them for every new token. The KV cache grows with prompt length and generated tokens, and it consumes GPU memory, which is why long contexts and long outputs raise memory requirements and can limit concurrency.

What is time to first token in LLM inference?

Time to first token (TTFT) is the delay from request arrival to the first generated token. It is dominated by the prefill phase, where the model processes the prompt and builds the KV cache, so TTFT scales with prompt length. TTFT determines whether a response feels responsive, and it is one of the two latency signals (alongside inter-token latency) that define the user experience of LLM serving.

What GPU memory does LLM inference use?

LLM inference uses GPU memory for the model weights, the KV cache, and activations. Weights are a fixed cost determined by model size; the KV cache grows with prompt length, generated tokens, and concurrency; activations depend on batch size. For large models or long contexts, the KV cache can exceed the weights, which is why memory management (capping context, evicting idle cache, paged attention) is a major inference optimization lever.

Summary

LLM inference is the phase where a trained model generates responses to prompts, running fixed weights forward to produce text one token at a time through prefill (processing the prompt) and decode (generating tokens). It differs from training in being memory-bandwidth-bound, continuous, and latency-sensitive rather than compute-bound, periodic, and throughput-tolerant. Inference is expensive because of GPU memory, compute per token, and utilization, and its cost compounds with scale in ways training does not. The infrastructure must provide enough memory and bandwidth for the model and KV cache, and techniques like batching and continuous batching are essential to keep GPUs saturated and cost per token low. Understanding inference is the foundation for every downstream topic — cost, latency, GPU choice, and serving architecture.

For teams moving LLMs into production, dedicated GPU infrastructure sized for inference provides the memory, bandwidth, and operational support that production serving demands.