Expert Parallelism Sends All-to-All, Not Another All-Reduce

A cluster can post a handsome all-reduce number and still be the wrong cluster for a mixture-of-experts run. Data parallel reduces: every rank holds a full copy of the same step and the fabric combines gradients in a pattern switches are now built to expect. Expert parallelism does something else. Each token is sent to the GPU that holds its expert and the result is sent back. That permutation is an all-to-all. It incasts. A sign-off built on a single ring test never saw it.

Two Collectives, Two Traffic Shapes

NCCL treats these as different operations because the wire pattern is different. Collapsing them into "the GPUs need a fast network" is how a purchase order gets written against the wrong test.

None of this says an MoE model is slower than a dense one. It says the network acceptance test for the MoE job is not the acceptance test you already ran for the dense job. If the slide in the deal only shows all-reduce bus bandwidth, ask what happens when every rank sends to every other rank at the message size your dispatcher actually uses.

Neighbor traffic makes the incast worse and is not the model's fault. On a shared Ethernet leaf, another tenant's burst lands in the same buffers as your permutation. A dedicated Spine-Leaf removes that neighbor. OneSource Cloud runs that dedicated RoCE v2 fabric for the tenant's own nodes: no second tenant on the NIC, PFC and ECN set for RDMA rather than left at best-effort defaults. The job's own all-to-all is still the job's. Dedicated does not mean the permutation got smaller.

Where an All-Reduce Fabric Feels It First

The symptom in the training log is dull. Step time rises. GPU utilization stays high, because the ranks are in the collective, which counts as busy. Nobody sees a CUDA error. The network half of that wait is usually one of two things operators already know from RoCE tuning and then forget when the model architecture changes.

• Hash polarization. ECMP picks a next hop from a hash of the packet. All-to-all flows are numerous and, if the hash inputs are poor, they pile onto the same links while neighbor links sit idle. The ring test used few flows and never showed it.
• Pause and mark pressure. A lossless RoCE profile reacts to that pile-up with PFC pause frames, or an aggressive ECN threshold cuts the rate harder than the burst required. Average counters sampled over seconds miss the microburst that triggered it. The collective, which is synchronized, does not miss it.

InfiniBand is not the requirement hiding inside this paragraph. A non-oversubscribed RoCE v2 path with jumbo MTU, a priority that matches the switch, and an all-to-all test is a requirement. Best-effort Ethernet with MTU 1500 is not that path, whatever the all-reduce slide said on a quiet afternoon.

Ask for an All-to-All Number Before You Sign

Put this in the capacity test, in writing, before the cluster is accepted.

1. Message sizes taken from the expert dispatch you will run, not from a 1 GB demo buffer.
2. A rank count at the expert-parallel width you will run, not at two nodes.
3. Link-utilization spread across the leaf, recorded during the test. A wide spread is polarization even if the average looks fine.
4. PFC pause frames and ECN marks over the same window. Flat is the pass condition. A test that only reports throughput can pass a fabric that is one burst from a stall.

What dedicated changes, and what it does not

Running that test on a single-tenant fabric tells you whether your job fits. Running it on a shared leaf tells you whether your job fits beside whoever else was busy that hour, which is a different question and a worse one to sign. OneSource Cloud's arrangement answers the second question by removing the other tenant. You still have to run the four lines above. A non-blocking network is not a substitute for an all-to-all at the real message size, and this page is not going to invent one.

Questions that belong in the test plan

Do I need InfiniBand to run expert parallelism?

No. You need a lossless path that is not oversubscribed, and you need to test all-to-all at the expert message size. RoCE v2 is that path when PFC, ECN, and MTU are set for it. A best-effort Ethernet that happened to pass a ring test is not. InfiniBand is sufficient for many of these jobs. It is not the definition of the requirement.

Does a OneSource Cloud cluster make expert-parallel all-to-all free?

No. OneSource Cloud keeps other tenants off the RoCE v2 fabric, so their incast is not mixed with yours. The tokens your own job permutes to its experts still cross the network, and that pattern still has to be measured at the dispatch size you will actually run. Dedicated removes a neighbor. It does not remove the collective.