Why GPUs Stall on NFS During Training: Diagnosis and Fixes

An NFS-induced training stall has a recognizable signature: GPU utilization sawtooths toward zero between steps while the model, the collective network, and the GPUs themselves are healthy. The cause is almost always the data path, and the fix is a ladder, not a purchase. This page gives the isolation test that proves NFS is the cause, the ordered diagnosis sequence from file shape to storage architecture, and the acceptance threshold that verifies the fix and keeps it from regressing.

Recognize the Stall Signature and Isolate the Cause

The signature of a data-path stall is visible before any tooling: GPU utilization sawtooths or drops toward zero between steps, step time inflates at epoch boundaries or after resume, and throughput improves when the dataset shrinks. Confirm with the two-part isolation test before touching configuration. First, run the identical training loop on synthetic data generated in memory: if throughput recovers to the hardware baseline, the bottleneck is the data path, because compute, NCCL, and the model are unchanged between the two runs. Second, measure per-worker read throughput during a stalled step (iostat or nfsiostat on the client): workers reading at a fraction of link speed while GPUs idle confirm storage-side starvation rather than collective-communication waits. The documented pattern matches: synthetic runs perform fine while real datasets bottleneck, and multi-node jobs degrade as nodes contend for one NFS head.

Watch for the two impostors that mimic the signature. A stalled step that coincides with checkpoint writes is checkpoint I/O, not dataset starvation, and it follows different fixes. A slowdown that appears only after a resume is often a cold cache, which the acceptance test later distinguishes from a genuine regression.

• Sawtooth GPU utilization with idle gaps between steps
• Step-time inflation at epoch boundaries or after resume
• Synthetic-data run recovering full throughput
• Per-worker reads at a fraction of link speed during the stall

The Diagnosis Sequence and Fix Ladder

Work the sequence in order and stop when the acceptance threshold passes. Step 1: quantify the gap as stalled throughput divided by synthetic-baseline throughput, which becomes the number every fix must move. Step 2: check file shape, because NFS punishes many small files; if the dataset is thousands of small files per epoch, repackage into shards (WebDataset or tar) so reads become sequential streams. Step 3: scale the data loader, raising worker count and prefetch factor until CPU or I/O saturates rather than GPUs waiting. Step 4: tune the NFS client with larger read-ahead and extended attribute caching (rsize, actimeo) for read-mostly training data. Step 5: stage hot data to local NVMe per node when datasets fit, converting the shared dependency into a warm cache. Step 6: when the gap persists at fleet scale, move the shared tier to parallel or high-throughput storage designed for aggregated reads. The ladder exists because each rung costs more than the last; skipping to storage purchases without steps 1 through 4 buys hardware for a problem that mount options would have solved.

1. Quantify the gap: stalled throughput divided by synthetic baseline.
2. Repackage small-file datasets into sequential shards.
3. Scale loader workers and prefetch until a real saturation point.
4. Tune NFS client read-ahead and attribute caching.
5. Stage hot data to per-node NVMe caches.
6. Move the shared tier to parallel high-throughput storage.

The economics of the ladder are the reason to respect its order. Steps 2 through 4 are configuration work measured in hours; step 5 is a caching layer; step 6 is architecture. Teams that reach step 6 should evaluate it as a design decision with measured numbers in hand, which is the position OneSource Cloud storage engagements start from: the tiered NVMe-oF fabric is sized against the measured gap, not against a catalog page.

Verify Against the Synthetic Baseline and Guard the Fix

The fix passes when stalled-config throughput reaches at least 90 percent of the synthetic-baseline throughput under the real dataset, measured across at least three epochs including a cold cache, because warm-cache numbers flatter any configuration. Verification then turns into a guard: export data-loader wait time per step and GPU utilization as training metrics, alert when utilization dips below the post-fix floor for more than a few minutes, and record the NFS server head load at that moment so the next regression distinguishes client regression from server contention. Two residual checks close the loop: re-run the synthetic baseline after any framework or driver change, since the baseline itself moves with the stack; and re-test at full fleet size, because single-node validation hides the multi-node contention that produced the original stall. A fix verified this way survives dataset growth; a fix verified by observation alone regresses at the next epoch boundary.

The guard has a second use beyond regression: the same data-loader wait metrics justify the step-6 architecture decision with evidence a finance review accepts, converting a performance complaint into a measured capacity argument.

• Acceptance: at least 90 percent of synthetic baseline across three epochs, one cold
• Guard one: data-loader wait time and GPU utilization exported and alerted
• Guard two: NFS head load recorded at every regression for attribution
• Re-baseline after any framework, driver, or fleet-size change

For teams that land on step 6, the architecture answer is a parallel tier: OneSource Cloud runs training data paths on tiered NVMe-oF parallel storage with GPUDirect Storage feeding single-tenant bare-metal GPUs, so aggregated reads distribute across parallel targets instead of serializing behind one NAS head.

FAQ

When should teams move training data off NFS entirely?

When the first four rungs of the ladder, sharding, loader scaling, mount tuning, and local staging, cannot close the gap to 90 percent of the synthetic baseline, or when multi-node growth keeps re-introducing the stall. At that point the shared tier itself is the constraint, and a parallel high-throughput tier is the fix that holds at fleet scale.

How does OneSource Cloud design storage so GPUs do not wait on data?

OneSource Cloud runs dataset reads on tiered NVMe-oF parallel storage with GPUDirect Storage zero-copy paths feeding single-tenant bare-metal GPUs, so aggregated reads land on parallel targets instead of one NAS head. Because the environment is dedicated and priced flat-rate monthly with zero egress fees, moving datasets into the parallel tier does not add transfer cost.

Summary: Prove the Cause, Climb the Ladder, Keep the Guard

The stall is cheap to prove and expensive to guess at: run the synthetic comparison, quantify the gap, and climb the fix ladder only as far as the 90 percent threshold demands. The regression guards keep the fix honest as datasets and fleet size grow. When the answer is architecture rather than tuning, OneSource Cloud delivers the parallel storage fabric that makes step 6 a measured decision instead of a leap.

To see the storage architecture that removes the NAS ceiling, explore OneSource Cloud AI storage architecture.