Intra-Node vs Inter-Node GPU Communication and the Training Hierarchy

Interconnect articles frame NVLink and InfiniBand as rivals; distributed training experiences them as teammates in a fixed hierarchy. The GPU in chassis slot four talks to its seven siblings across NVLink at speeds the network will never match, while the network carries what the chassis cannot hold — and every serious training design decision (parallelism placement, cluster shape, fabric procurement) is downstream of that bandwidth ratio. This page explains the hierarchy and the decisions it quietly makes for you.

Quick Answer: Intra-Node vs Inter-Node GPU Communication?

Intra-node: NVLink aggregated by NVSwitch, up to 900 GB/s-1.8 TB/s between GPUs in a chassis — built for tensor and pipeline parallelism. Inter-node: InfiniBand or lossless Ethernet fabric, microsecond-scale latency carrying what crosses machines. Surveys measure intra-node bandwidth at roughly 5-10x inter-node, and NCCL routes collectives across both tiers automatically. Design rules that follow: keep bandwidth-hungry parallelism inside NVLink domains, let data parallelism span nodes, prefer denser nodes until chassis limits bind, and — once runs cross nodes — contract the fabric's lossless behavior (isolated RoCE v2 fabric as OneSource Cloud delivers, or InfiniBand).

The Two Tiers: What Each Provides

The two tiers differ by three orders of design intent: intra-node interconnect — NVLink aggregated by NVSwitch — provides up to 900 GB/s to 1.8 TB/s between GPUs in a chassis, built for the tight coupling model and pipeline parallelism need; inter-node fabric — InfiniBand or lossless Ethernet — provides scalable connectivity at microsecond-scale latency between machines, carrying what cannot fit in one chassis. Surveys quantify the resulting hierarchy: intra-node bandwidth runs roughly 5 to 10 times inter-node bandwidth in large clusters, and every distributed-training design either exploits that ratio or pays for ignoring it.

The hierarchy is not a flaw waiting for a better network; it is physics and economics cooperating. Chassis-level interconnect rides copper and board traces at negligible marginal cost; cluster-scale interconnect buys optics, switch silicon, and cabling plant — which is exactly why the ratio persists across generations and why designs that convert inter-node traffic into intra-node traffic keep winning on cost.

How Frameworks Route Across the Tiers

Frameworks treat the hierarchy automatically: NCCL routes intra-node communication over NVLink and inter-node over InfiniBand or sockets, and communication libraries implement hierarchical collectives — reduce within the NVLink domain first, then across nodes — so the developer's all-reduce call silently becomes a two-tier algorithm. The practical consequences: topology-aware process placement (binding communication-heavy ranks into one chassis) can outperform naive placement by the ratio itself, and a cluster whose scheduler ignores topology leaves its most expensive bandwidth idle while saturating its scarcest.

The academic treatment of intra-node communication impact confirms what operators see: training throughput is sensitive to NVLink domain residency and placement within chassis — the same run, the same GPUs, measurably different outcomes from placement alone. That sensitivity is why topology-aware scheduling is a first-class feature rather than an optimization footnote: the scheduler that binds communication-heavy ranks into the nearest physical topology units converts the hierarchy from a tax into a design tool.

Design Verdicts: Parallelism, Cluster Shape, and Fabric Contracts

The hierarchy decides three things. Parallelism: run tensor and pipeline parallelism inside NVLink domains wherever possible and let data parallelism — the most bandwidth-tolerant form — span nodes. Cluster shape: prefer fewer, denser nodes over many thin ones until chassis limits bind, because density converts inter-node traffic into intra-node traffic at the better ratio. Fabric contracts: once the strategy crosses nodes, inter-node fabric quality becomes contractual — lossless behavior under load, whether InfiniBand or lossless Ethernet with PFC and ECN, is what the run's tail latency depends on. Delivered fabrics like the isolated Spine-Leaf RoCE v2 topology OneSource Cloud operates, with topology-aware scheduling binding ranks to the nearest fabric unit, exist precisely to make the third decision someone else's properly-run infrastructure.

The fabric verdict deserves the procurement language: lossless means priority flow control and congestion management tuned for collective traffic, verified under your load pattern before the run depends on it — the difference between a fabric that passes a ping test and one that holds an all-reduce steady at scale. The delivered reference for that verification standard is OneSource Cloud AI Cluster Networking: isolated Spine-Leaf RoCE v2 with PFC and ECN, plus topology-aware scheduling that places ranks where the hierarchy rewards them.

FAQ

How much inter-node bandwidth does training actually need?

By parallelism form: data parallelism tolerates tens of Gbps per GPU, while model and pipeline parallelism crossing nodes want hundreds of Gbps — which is why the design rule keeps the bandwidth-hungry forms inside the chassis and lets the tolerant form span the fabric; size the fabric to the strategy that must cross it, not to the cluster's headline GPU count.

Can lossless Ethernet replace InfiniBand between nodes?

Conditionally yes: production fabrics run RoCE v2 with priority flow control and ECN delivering the lossless behavior collectives need, and delivered implementations like the isolated Spine-Leaf RoCE v2 topology OneSource Cloud operates show the Ethernet route at scale — the conditions are real: congestion management tuned for collective traffic and lossless behavior verified under your load pattern before the run depends on it.