Gang Scheduling Still Fails If It Cannot See NVLink

Gang scheduling has one job that people remember: do not start seven ranks of an eight-GPU collective while the eighth is still queued, or the seven will sit in a barrier until the job is killed. That rule is necessary. It is not the rule that decides whether the collective is fast. Eight GPUs that start together and sit on three different NVLink domains will pass the gang check and then spend the run waiting on a path that was the wrong path. "All or nothing" never said "all of them nearby."

All-or-Nothing Is Not the Same as Nearby

Upstream Kubernetes scheduler work treats topology as a score you can add. That is the tell. If nearby placement were part of gang scheduling, it would not be a separate plugin. Default gang behavior promises a start time. It does not read which GPUs share a bridge.

You want both. A topology-aware policy that starts ranks one at a time reintroduces the deadlock gang scheduling was invented to kill. A gang policy that treats any eight free GPUs as a match reintroduces the straggler that is really a long wire. Teams who "turned gang scheduling on" and still see one slow rank have usually bought only the first row.

The second row needs a scheduler that is allowed to see the machine. On dedicated nodes the machine is stable: the NVLink groups do not change because another tenant was placed beside you. OnePlus, the orchestration layer OneSource Cloud runs on those nodes, uses that stability. It places a communication-heavy gang inside the nearest NVLink domain and the same rack before it will cross a spine on the RoCE v2 fabric. A default gang scheduler will not. It will take the eight free devices the bin-packer found, and the collective will be correct and slow.

The Four Facts a Placement Needs

If the policy cannot answer these, it is guessing. NVIDIA's GPU operator already exports device topology into the cluster on a normal install. The failure is usually that nothing reads it.

1. NVLink domain. Which GPUs share the high-bandwidth bridge. A collective that fits inside one domain should not leave it.
2. PCIe root. Which GPUs hang off the same root complex. Leaving the NVLink domain for a neighboring PCIe hop is already a different latency, and jobs that "only" did that will not show a fault.
3. Rack. Which GPUs share a top-of-rack switch. Crossing a rack crosses a spine, even on a non-blocking fabric, and that hop belongs to jobs that do not fit in one rack. It should not be the default for an 8-GPU gang.
4. The path between racks. When the gang genuinely does not fit, the scheduler should know which racks share a spine, instead of treating every remote GPU as the same distance.

What OnePlus does with the four

It prefers the smallest domain that can hold the gang. Communication-heavy ranks stay on the NVLink bridge they can actually saturate. The RoCE hop is reserved for the job that asked for more GPUs than one domain holds. That is a placement rule, not a claim that every job becomes network-bound or that every job fits on one node. A job bigger than a rack still crosses a spine. The difference is that the crossing is a decision, visible in the placement, rather than an accident of which cards were free.

Prove It by Pinning, Not by Staring at Utilization

Utilization will not catch this. Ranks waiting in a collective look busy. The test is two runs of the same job, same batch, same number of GPUs.

• Run A: pin every rank to GPUs you have confirmed share one NVLink domain. Record step time.
• Run B: pin them so the gang crosses domains or racks, the way an indifferent bin-packer would. Record step time.
• If only B is slow, the scheduler policy is the bug. Fix the policy or keep the pin. Do not retune the learning rate.
• If both are slow, stop blaming placement. You have a different bottleneck, and another afternoon of affinity masks will not find it.

A pin is a legitimate workaround for one team and a bad plan for a cluster. The moment a second group submits an 8-GPU job, the human who knew which GPUs share a bridge is not in the loop. The policy has to know. That is the entire requirement this page adds to gang scheduling, and it is testable in an afternoon.

Questions after the first slow collective

Can I fix a bad placement without changing schedulers?

For one job, yes. Pin it to a node set you have checked is a single NVLink domain, and keep that set documented. The pin does not survive the next team, the next node replacement, or a job that no longer fits the set. Those are the cases the scheduler has to handle by reading topology rather than by inheriting someone's notes.

What does OnePlus know that a default gang scheduler does not?

OnePlus places the gang inside the nearest NVLink domain and the same rack, and only then crosses a spine on the dedicated RoCE fabric. A default gang scheduler promises the ranks start together. It does not know which GPUs share a bridge, so "together" can mean three racks. The collective still completes. It completes at the speed of the longest hop the placer happened to choose.