How Much GPU Cluster Bandwidth AI Training Actually Needs

Ask how much bandwidth a training cluster needs and the answers scatter: vendor spec sheets quote NVLink aggregates in terabits, practitioner rules say hundreds of gigabits per GPU, and procurement documents end up with a number nobody derived. The derivation exists and it is not hard — your parallelism strategy determines what traffic the network carries, the collective structure determines its shape, and the requirement falls out as arithmetic. This page does the arithmetic.

Quick Answer: How Much Bandwidth Does AI Training Need?

Derive, don't quote: data parallelism ships gradients between steps and tolerates tens of Gbps per GPU; model and pipeline parallelism ship activations every pass and want hundreds of Gbps — practitioner rules land inter-node needs at 200-400 Gbps per GPU for communication-heavy training. The tiers: intra-node NVLink at 3,600 Gbps aggregate per H100 (900 GB/s bidirectional via NVSwitch), inter-node at 400 Gbps-class links (NDR-class fabrics). Every figure assumes lossless behavior — one dropped packet can stall every GPU in the job. Procure on sustained all-reduce throughput under load, not line-rate specs.

Definition: What Sets the Requirement

The parallelism strategy sets the requirement, not the GPU count: data parallelism ships gradients between steps and tolerates tens of Gbps per GPU; model and pipeline parallelism ship activations inside every forward and backward pass and want hundreds of Gbps — which is why practitioner guidance lands inter-node needs at 200 to 400 Gbps per GPU for communication-heavy training while tolerant forms run on far less. The collective pattern adds its shape: all-reduce dominates data parallelism (ring patterns, bandwidth-optimal), while model parallelism's point-to-point traffic follows the layer structure — the derivation starts by naming which forms your run uses and what each ships.

The derivation's first move is naming the mix: most real training runs combine forms — data parallel across replicas, model parallel inside them — and the inter-node requirement follows the hungriest form that crosses nodes. Keep the bandwidth-hungry forms inside the chassis where the fast tier lives, and the fabric requirement relaxes; let them cross nodes, and the fabric budget is set by the derivation's answer, not by what the quote happened to include.

Enterprise Relevance: The Two Tiers with Real Numbers

The requirement lands on two tiers with an order of magnitude between them: intra-node, where NVLink's fourth generation provides 3,600 Gbps aggregate per H100 (NVSwitch delivering 900 GB/s bidirectional per GPU) — the tier model parallelism lives on; and inter-node, where 400 Gbps NDR links carry what crosses machines at roughly one-ninth the per-GPU bandwidth, with 10,000-GPU production clusters built on exactly that link class and all-reduce achieving a fraction of theoretical peak in practice. The budget asymmetry follows: the intra-node tier ships with the servers, while the inter-node tier is the fabric line item — which is why the derivation's verdict about how much traffic crosses nodes decides most of the network cost.

The real-world footnote belongs in the budget conversation: production clusters on 400G fabrics achieve a fraction of theoretical peak on all-reduce-bound workloads — the fraction is congestion, topology, and straggler physics, which is why the acceptance metric in the next section is sustained throughput under your load, never the link's paper rate.

Boundary: Losslessness and the Procurement Translation

Every bandwidth figure assumes lossless behavior: a single dropped packet can stall every GPU in the training job, because collectives wait for their slowest participant — so the requirement is never bandwidth alone but bandwidth plus the congestion management (PFC and ECN-class flow control) that keeps collective traffic inside it. In procurement language: state the per-GPU inter-node bandwidth by parallelism form, require sustained all-reduce throughput at that rate under load rather than line-rate ping tests, and write the lossless behavior into the fabric acceptance — the translation that separates fabrics meeting spec sheets from fabrics meeting training. Delivered fabrics make the translation inspectable: the isolated Spine-Leaf RoCE v2 topologies OneSource Cloud operates publish their per-GPU paths and verify lossless behavior under collective load before campaigns depend on them.

The translation is the difference between buying bandwidth and buying training throughput — spec-sheet fabrics pass ping tests and fail collectives, while acceptance-language fabrics demonstrate the number your derivation produced under the traffic your training generates. The delivered reference for that standard is OneSource Cloud AI Cluster Networking: published topology, lossless RoCE v2 behavior verified under collective load.

FAQ

Is 100G Ethernet ever enough for GPU training?

For tolerant forms at small scale, sometimes: data-parallel training with gradient accumulation and small models can hold step times on 100G inter-node links — the derivation decides, and if your communication volume per step fits the link with acceptable step-time overhead, 100G serves; the moment model or pipeline parallelism crosses nodes, the requirement moves to hundreds of Gbps and 100G becomes the bottleneck you modeled incorrectly.

What does lossless mean operationally for a training fabric?

That collectives complete without retransmission stalls at the transport layer: priority flow control prevents drops on congested links, explicit congestion notification signals slowdowns before drops, and the fabric sustains all-reduce throughput at the contracted rate under load — the properties OneSource Cloud verifies on its RoCE v2 fabrics before campaigns run, because bandwidth without losslessness is a number your training will not experience.