Per-GPU NIC Sizing for AI Cluster Networks

Cluster network design fails in a predictable way: someone copies a reference server, halves the NICs to save cost, and discovers during the first large training run that collectives stall and GPUs idle. Per-GPU NIC sizing prevents that failure by deriving network requirements from your workload before procurement, then validating the built fabric against the same numbers. This guide provides the ledger method, the one-NIC-per-GPU rule with its legitimate exceptions, and the validation steps that prove the fabric delivers.

Start From the Traffic Ledger, Not the Spec Sheet

Build a per-GPU ledger with three rows: collective traffic for distributed training (driven by all-reduce volume and the sync cadence of your parallelism scheme), data-path traffic for storage and checkpointing (GPUDirect or staged), and ingress/egress for serving or ingestion. The sum, with headroom, is the per-GPU bandwidth requirement that NIC count and speed must meet, and reference designs exist to anchor the arithmetic.

Fill the three-row ledger per GPU before touching any vendor datasheet:

Add headroom for bursts and stragglers, and the ledger sum becomes the per-GPU bandwidth requirement your NIC count and speed must meet. The reference anchors keep the arithmetic honest: the previous A100 generation used the same 1:1 ratio at 200 Gb/s, so the pattern is structural, not a fashion of one product cycle.

Applying the One-NIC-Per-GPU Rule and Its Exceptions

Match one high-speed NIC to each GPU whenever collectives or GPUDirect traffic dominate the ledger, because GPUDirect RDMA maps NICs directly to GPU memory and a shared NIC becomes the per-GPU ceiling. Bend the ratio only where traffic classes separate cleanly: storage and management run on their own lower-speed interfaces in reference designs, and inference-only clusters with modest ingress can aggregate two GPUs per fabric endpoint when the ledger proves it.

1. Compare ledger to endpoint. When collectives or GPUDirect traffic dominate, assign one high-speed NIC per GPU; GPUDirect RDMA maps the NIC directly to GPU memory, so a shared NIC becomes a per-GPU ceiling that no software tuning removes.
2. Separate traffic classes. Keep storage and management on their own lower-speed interfaces as reference designs do; mixing them into the compute fabric borrows bandwidth your collectives need.
3. Bend the ratio only with proof. Inference-only clusters with modest ingress can aggregate two GPUs per endpoint when the ledger shows the aggregate fits with headroom; document the ledger, not just the conclusion.
4. Check the fabric, not just the NIC. A rail-optimized topology gives each GPU NIC a dedicated rail so the endpoint bandwidth is real at cluster scale; an oversubscribed spine erases the NIC investment.

Production fabrics show what the ledger looks like when it is engineered end to end: OneSource Cloud runs a non-blocking Spine-Leaf RoCE v2 network with hardware offload and lossless priority flow control, giving each GPU a deterministic share of fabric bandwidth for collective operations and GPUDirect transfers instead of an oversubscribed slice.

Validation: Proving the Fabric Delivers Per-GPU Bandwidth

Validate with load, not link lights: run collective benchmarks at cluster scale and confirm bisection bandwidth under all-to-all stress, verify lossless behavior under congestion (priority flow control and congestion notification doing their job), and confirm GPUDirect paths sustain ledger rates end to end. A fabric that passes single-stream tests but collapses under collective load is oversubscribed somewhere the ledger will find.

1. Run collective benchmarks (all-reduce, all-to-all) at production scale, not two-node samples, and compare achieved bandwidth to the ledger.
2. Stress congestion behavior: lossless fabrics must show priority flow control and congestion notification handling load without collapse.
3. Validate GPUDirect paths end to end at ledger rates, because a fast NIC behind a slow PCIe path or storage tier is not a fast NIC.

Map every validation result back to its ledger row. When a row underperforms, the fabric, host, or storage path will identify itself, which is the point of keeping the ledger: it converts a vague network complaint into a named, fixable gap.

FAQ

Does inference serving need one NIC per GPU like training does?

It depends on the ledger, not the workload label. Training collectives and GPUDirect checkpointing usually justify one NIC per GPU, while inference clusters with modest ingress can sometimes aggregate two GPUs per endpoint when the ledger shows the aggregate fits with headroom. The discipline is writing the ledger down before the design review, not the ratio itself.

How does OneSource network design handle per-GPU bandwidth?

OneSource runs a non-blocking Spine-Leaf RoCE v2 fabric with hardware offload and lossless priority flow control, so each GPU receives a deterministic bandwidth share sized for collective operations and GPUDirect traffic. The per-GPU guarantee is a fabric property, not a per-server spec, which is what keeps cluster-scale validation results matching the design.