Running Distributed LLM Inference Across Multiple GPUs

Quick Answer: Distributed LLM inference splits a model whose weights and working state exceed a single GPU across multiple GPUs, using tensor parallelism to divide each layer's computation, pipeline parallelism to assign layer groups to different GPUs, or a combination of both. The strategy choice trades communication volume against utilization, and the interconnect between GPUs frequently determines real-world performance more than the GPUs themselves. Teams that match the parallelism strategy to their model shape and latency targets serve large models efficiently; teams that guess usually buy hardware that underperforms its price.

The need is now routine. Memory-rich GPUs serve many mid-size models single-handedly, but large dense models, high-throughput serving farms, and long-context workloads all push past what one accelerator can hold. Distribution is the answer, and its engineering details decide whether the result is an efficient serving platform or an expensive lesson in communication overhead.

Why Models Outgrow a Single GPU

Distributed inference is the practice of partitioning a model's weights and execution across multiple GPUs so that serving can proceed when the full model plus per-request state does not fit in one device's memory. The arithmetic is unforgiving: model weights, plus the KV cache that grows with concurrent requests, plus activation memory, must all fit somewhere. When that somewhere exceeds one GPU, the model is sharded.

Sharding choices differ in what they divide. Tensor parallelism splits individual layers, placing fractions of each weight matrix on different GPUs that exchange intermediate results every layer. Pipeline parallelism splits the model into sequential stage groups, with each GPU or group handling a contiguous slice of layers and activations flowing between stages. Data parallelism replicates the whole model for higher aggregate throughput, which addresses request volume rather than model size.

Tensor Parallelism: Fast but Communication-Hungry

Tensor parallelism keeps every GPU participating in every request, which yields low latency for individual tokens: all shards compute simultaneously, and each layer's outputs are combined through fast collective operations. The cost is communication volume, because those collectives run between layers, and the speed of the interconnect between GPUs directly gates how much of the theoretical speedup survives.

This is why intra-node placement dominates tensor-parallel designs. GPUs within a server share high-bandwidth links such as NVLink, with latencies measured at scales that keep per-layer collectives affordable. Spreading tensor parallelism across nodes, over cluster networking, multiplies per-layer communication cost and usually degrades latency beyond what the added GPUs contribute. The practical rule: tensor-parallel degree should stay within a single well-connected node.

Pipeline Parallelism: Bandwidth-Friendly but Latency-Prone

Pipeline parallelism communicates only at stage boundaries, sending activations forward between layer groups. Its communication volume is far lower per request, which suits inter-node distribution where network bandwidth and latency are the constraints. The trade-offs are utilization bubbles, because stages idle while waiting for adjacent stages on a single request, and added request latency from the sequential hand-offs.

Continuous batching mitigates the bubbles by keeping every stage busy with different requests in flight, which makes pipelining effective at meaningful concurrency but awkward for very low-latency, low-traffic endpoints. Production architectures commonly combine the two: tensor parallelism within a node for speed, pipelining across nodes for capacity.

Interconnect Requirements: The Hidden Decider

Serving performance of distributed models tracks interconnect performance with a directness that surprises teams used to single-GPU work. Intra-node, NVLink-class bandwidth keeps tensor-parallel collectives fast. Inter-node, the cluster fabric, whether InfiniBand or high-speed Ethernet with RDMA, sets how much pipelining and cross-node communication cost. A fleet with strong GPUs and weak networking will underperform a smaller fleet with balanced design.

Latency budgets make this concrete: every cross-GPU exchange in the decode path adds delay that accumulates per token and per layer boundary. Providers that engineer the full path, such as high-performance AI networking designs in dedicated environments, treat interconnect as a first-class design input rather than an afterthought.

Sizing and Operating a Distributed Serving Fleet

Sizing follows from model shape and service targets rather than from GPU count ambitions. Start from model size plus planned KV cache per concurrent request to determine minimum shard footprint. Then apply the placement rules: tensor-parallel degree bounded by node topology, pipeline degree added when capacity requires more nodes, and data-parallel replicas added for throughput once a serving unit is defined. Validate with realistic traffic, because concurrency changes which parallelism configuration wins.

Operations add their own requirements. Health monitoring must cover the interconnect, not just GPUs, since a degraded link manifests as mysterious latency rather than a clean failure. Deployments and rollbacks touch every shard, so automation matters more than in single-GPU serving. And capacity planning must account for cache growth under traffic peaks, which is where under-provisioned fleets fail first.

FAQ

How many GPUs does it take to serve a 70B model?

It depends on precision and serving targets. At half precision, weights alone approach the limits of one 80GB-class GPU before any cache, so practical serving uses at least several such GPUs, with the exact count set by concurrency and context-length targets rather than weights alone. Quantization reduces the floor but not to zero cache pressure.

Tensor or pipeline parallelism: which should I use?

Use tensor parallelism within a node for low-latency serving, because per-layer collectives need NVLink-class bandwidth. Add pipeline parallelism across nodes when capacity demands more hardware than one node provides. Most production deployments combine both, plus replicas for aggregate throughput.

Why is networking so important for distributed inference?

Every layer boundary in tensor parallelism and every stage boundary in pipelining crosses an interconnect, and decode repeats those crossings for each token generated. Weak networking turns fast GPUs into a slow fleet, so interconnect bandwidth and latency belong in the serving design from the start.

Does distributed inference increase latency?

Relative to a hypothetical single GPU holding the whole model, yes, because communication steps add time. Relative to the realistic alternative of not serving the model at all, well-placed parallelism keeps latency acceptable: intra-node tensor parallelism adds little, while long pipeline chains add more.

What should I monitor in a distributed serving deployment?

Monitor per-shard GPU health, interconnect errors and saturation, per-stage queue depths, and end-to-end latency decomposition. Distributed failures often appear as partial degradation, such as one slow shard straggling the rest, so per-component visibility matters more than aggregate dashboards.

Summary

Distributed LLM inference is the standard answer for models beyond a single GPU, with tensor parallelism providing speed within nodes, pipeline parallelism providing capacity across them, and interconnect quality deciding how much performance survives. Size from model shape plus cache targets, respect node boundaries in placement, and operate with per-component monitoring. Teams that treat networking as a first-class input get serving fleets that match their hardware investment.

To design a serving architecture for your large models, request an architecture review from OneSource Cloud, or explore private AI infrastructure engineered for multi-node model serving.