Read the NCCL Timeout Before You Shrink the Batch

A line that says the NCCL watchdog caught a collective timeout is a label, not a diagnosis. The same string covers a job that never finished rendezvous, a single rank that is late to every barrier, a RoCE fabric that paused itself, and a cluster that silently fell back to TCP sockets. Cutting the global batch changes the offered load. It does not tell you which of those four you have, and on three of them it makes the next incident quieter while the path stays broken.

Four Timeouts That Share One Log Line

Open the NCCL log before you open the training config. Set NCCL_DEBUG=INFO on a reproduction, then sort the incident into one of four buckets. The buckets do not share a fix.

Rendezvous failures are usually a socket interface, a firewall, or a wrong NCCL_SOCKET_IFNAME. They look dramatic and they are the cheap ones. The expensive miss is the fallback: RDMA initialization failed, NCCL kept running on TCP, and the only clue is a short clause in the info log. A bandwidth number at half of the NIC, or worse, is the hint to grep for that clause before anyone retunes a model.

Shared multi-tenant Ethernet makes the congestion bucket more likely, because another tenant's burst is your pause frame. A dedicated fabric does not make the other three buckets impossible. OneSource Cloud places a training pod on a single-tenant Spine-Leaf RoCE v2 network, so a neighbor is not on the NIC. Image skew, a bad GID index, and a sick local GPU still produce the same watchdog line on that pod.

Counters That Separate a Stall from a Straggler

Once the log says the collective started, stop reading the training loss. It will not move, which you already know. The next measurement decides the owner.

1. On every host, confirm the NIC the process actually opened. NCCL_IB_HCA and the info log should name the fabric HCA, in NUMA order with the GPUs, not a management interface.
2. Compare driver, CUDA, NCCL, and NIC firmware across the allocation. One drifted node is enough to wedge a collective or to pass it slowly.
3. Read PFC pause-frame counters and ECN marks on the NICs, and ask the fabric owner for the same counters on the top-of-rack ports for the same minute. Marks without pause frames point at congestion control being too aggressive. Pause frames that climb together point at a lossless config that has deadlocked or run out of headroom.
4. Pull the slowest host out and run a single-node NCCL test. If it is slow alone, the fabric is not the cause. Clocks, ECC, a negotiated PCIe width, or a thermal cap belong to that host.
5. If the single-node test is clean and the counters moved with the hang, stay on the fabric. MTU left at 1500, a DSCP mark that does not match the priority the switch is watching, and a GID index that points at the wrong address family are all in this branch.

What you are allowed to change while you look

Raising NCCL_IB_TIMEOUT is a measurement, not a repair. A longer timer tells you whether the collective would have finished. It does not make a paused priority group unpause. Leave the global batch where the runbook says it belongs until step 4 has cleared the straggler branch.

On a RoCE v2 Spine-Leaf, those pause and ECN settings are a property of the switches and the NIC profile, not a sysctl a training container should be inventing per job. That is the part a host such as OneSource Cloud owns: the lossless profile is applied once for the dedicated pod, and a tenant workload does not get to retune PFC because a neighbor's job changed shape. The tenant still owns the image pin and the test in step 4.

Why a Smaller Batch Hides the Fault

Collectives fail under the message size and the injection rate the job actually uses. A smaller batch shortens messages and lowers that rate. A lossy or barely-lossless path can then complete, which is how a "fixed" config reaches the next, larger run and times out again. The verification is a collective test at the original message size, on the original rank count, after the change you claim is the fix.

• Pass: the all-reduce or all-to-all holds bus bandwidth near the NIC's expected fraction, pause-frame counters stay flat, and no rank's step time drifts across a five-minute window.
• Fail: bandwidth collapses as message size grows, or one host's test is an outlier when run alone. Go back to the bucket. Do not ship the smaller batch as the mitigation.
• The one legitimate batch change: a rank actually runs out of GPU memory. That is an OOM in the log, not a watchdog line. Treat it as a memory bug.

Goodput is the number worth writing down. A job that "completes" on TCP at a third of the fabric rate has not been repaired. It has been moved to a path that will not show up in a GPU utilization chart.

Questions people ask after the first timeout

Is raising NCCL_IB_TIMEOUT a fix?

No. A longer timeout waits longer on the same stall, and a stall that is a PFC deadlock will not finish at any timer you are willing to set. Use a longer timer once, on a reproduction, to separate a slow collective from a dead one. Then repair the path the log and the counters named: the interface, the image pin, the host, or the lossless profile.

Can OneSource Cloud remove NCCL timeouts caused by a shared Ethernet fabric?

OneSource Cloud places training nodes on a single-tenant RoCE v2 Spine-Leaf fabric, so a neighbor tenant cannot inject the pause frames that stall a collective. That removes one bucket. It does not remove a rendezvous bug, a silent fallback to TCP, a drifted NCCL version, or a GPU that is slow when you test it by itself.