How to Tell If GPU Training Is Storage-Bound vs Network-Bound

GPU training is storage-bound when accelerators wait on reads, opens, or checkpoint writes. It is network-bound when they wait on collectives or parameter exchange between nodes. Those two waits look similar on a “GPU busy” chart and need different fixes.

Storage-bound vs network-bound GPU training is a diagnosis that locates idle time in the data path or in the collective path before you change filesystems or fabrics. You tell them apart with a step-time breakdown, not with a vendor preference.

This is a how-to for reading that breakdown. It is not an InfiniBand versus Ethernet comparison and not a treatise on undersized training networks.

Which signals should you collect on one representative job?

Collect these on the same job, same batch size, and same cache state. A cold first epoch is almost always storage-heavy. A warm epoch that still stalls on NCCL is a different ticket. Do not mix those traces.

How do you run the diagnosis without changing the cluster?

Pick one training job that represents production: same model width, same sequence length, same checkpoint interval. Record wall time per step, time in data load, time in forward/backward, time in optimizer, and time in collectives. Most frameworks expose this. If yours does not, add timers before you buy hardware.

Run three short trials: cold cache, warm cache, and checkpoint-on versus checkpoint-off. If warm cache plus checkpoint-off still shows GPU idle, look at collectives and host sync. If only cold cache or checkpoint-on explodes step time, stay on the storage ticket.

Watch the storage NIC and the training fabric NIC as separate objects. Many nodes have both. A saturated storage path with a quiet training fabric is not a fabric problem. The reverse is not a Lustre problem. Photograph both graphs for the same timestamps.

What should you change after the diagnosis?

If storage-bound, fix dataset layout, metadata pressure, read-ahead, local cache, or checkpoint frequency before you order a new filesystem. Tiny files and per-sample opens idle expensive GPUs. Shard files and stop treating scratch as a million-inode tree.

If network-bound, fix batching of collectives, overlap of compute and communication, or the job’s parallel topology before you relabel the fabric. A single fat allreduce on an otherwise healthy network can still dominate. Topology and schedule belong in the job, not only in the switch.

Do not “try RoCE versus InfiniBand” as the first experiment. That comparison answers a different question and will not help a dataloader that cannot open files. Keep fabric selection on a networking design page after this diagnosis is closed.

On dedicated halls, you can keep storage and training fabrics observable as first-class paths. AI storage architecture and AI networking are the two follow-on surfaces once you know which wait dominates. OneSource Cloud private AI infrastructure is a fit when you need those paths designed together on U.S. dedicated GPUs. It is not a substitute for the timers.

FAQ

Can a job be bound by both in one step?

Yes. A step can wait on data, then wait on allreduce. Report both percentages. Name the larger term as the primary ticket so you do not open two change windows. Re-measure after the first fix. The secondary wait often becomes visible only then.

Does GPUDirect Storage prove the job is storage-bound?

No. GDS is a possible acceleration for a storage path you have already shown is hot. Enabling GDS on a network-bound job wastes a change window. Prove the wait, then evaluate GDS as a tactic.

Why is first-epoch time a bad single metric?

First epoch fills caches and walks metadata. It almost always looks storage-bound. Production pain is often the warm steady state or the checkpoint cadence. Use first epoch as one trial, not as the only SLA.

Should we add GPUs to test the bottleneck?

A controlled scale-out test helps. If step time grows with nodes while local I/O stays flat, you are moving toward network-bound. If more readers melt the metadata server, you are still storage-bound. Change one axis. Do not add GPUs and a new filesystem on the same night.

Where do managed operations help?

They help when someone must keep the two NIC graphs and the job timers aligned after hours. That is an operations product, not a diagnosis method. Review managed AI infrastructure only after you know which wait you want staffed.

Summary

Storage-bound training waits on data and checkpoints. Network-bound training waits on collectives. Split the step, run cold/warm/checkpoint trials, and watch storage NICs separately from training NICs. Do not start with a fabric brand fight.

Once the wait is named, take the work to storage design or networking design, and keep dedicated GPU environments honest with the same breakdown on every change ticket.