GPU Cluster Networking Requirements for Distributed AI Workloads

GPU cluster networking must deliver the bandwidth, latency, and topology to handle collective operations (all-reduce, all-gather), east-west traffic between GPUs, and multi-node model parallelism — and networking inadequacy is the most common reason clusters underperform despite enough GPUs. For the evaluation methodology, see evaluating AI cluster networking. For interconnect specifics, see NVLink vs InfiniBand.

Requirements by Workload Type

Distributed training demands high bandwidth for collective operations at scale — all-reduce runs on every training step, and its bandwidth requirement grows with GPU count. The fabric must sustain this bandwidth under concurrent load without congestion, because a collective operation waits on the slowest link. Latency matters less for training than for inference, but communication overhead caps scaling. For the training sizing, see how many GPUs for LLM training.

Multi-node model serving demands low latency rather than peak bandwidth — each forward pass may cross the network for tensor-parallel inference, and the latency of those crossings adds to per-token generation time. Jitter (latency variance) matters as much as baseline latency. For the inference latency networking, see low latency networking for inference. Mixed workloads need a fabric that handles both profiles — the sustained bandwidth of training and the low-latency, low-jitter demands of inference — simultaneously without one degrading the other.

Design Requirements

Topology: a flat, non-oversubscribed fabric where GPUs that communicate frequently are one hop apart minimizes both latency and congestion. Oversubscription (more GPUs sharing a link than the link can serve at peak) produces congestion and tail latency. Bandwidth: sized to the peak collective operation demand at the cluster's scale, with headroom for concurrent workloads. Latency and jitter: low baseline latency plus low variance under load, tested at the cluster's scale with concurrent traffic. Protocol: RDMA (InfiniBand or high-speed Ethernet) is the standard for GPU clusters; TCP adds latency that AI workloads cannot afford at scale. For the testing methodology, see evaluating AI cluster networking.

FAQ

What networking does a GPU cluster need?

A non-oversubscribed fabric with RDMA, flat topology, bandwidth sized for collective operations at scale, and low latency plus low jitter under concurrent load. The requirements differ by workload: training needs bandwidth, inference needs low latency and jitter. See the design requirements above.

How does networking bottleneck GPU cluster performance?

When the fabric cannot sustain the bandwidth of collective operations (training) or introduces latency and jitter on every forward pass (inference), GPUs idle waiting on communication. The symptom is low model FLOPs utilization — GPUs are fast but the network is slow, so the GPUs wait. For how to test for this, see cluster networking evaluation.

Summary

GPU cluster networking requirements are bandwidth for training, latency and jitter for inference, and a non-oversubscribed RDMA fabric for both. For the full evaluation and design framework, see evaluating AI cluster networking.