Dataloader Stall vs Compute Stall in GPU Training

Quick Verdict: If the GPU is waiting, fix the input path. If the GPU is busy and the step is still slow, fix the math, the communication, or the batch shape. Calling every delay “storage” hides a compute stall. Calling every delay “the model” hides a dataloader stall.

A dataloader stall is idle GPU time spent waiting for the next batch, while a compute stall is time spent inside device kernels or collectives after the batch has already arrived. Both lengthen step time. Only one of them shows up as an empty SM.

This page is a diagnosis split. It is not a storage-throughput essay about GPUs waiting on a filesystem, and not an NFS postmortem. Those pages name a medium. This one names which side of next(batch) you are on.

What does each stall look like on a trace?

A dataloader stall is a gap before the first kernel of the step. Host CPU, workers, decode, or a queue into pinned memory is late. Utilization drops. Power may drop with it. A compute stall is a long kernel, a fat backward, or a collective that holds the devices while the batch sits ready.

Average GPU utilization is a poor headline. A job can show 80% and still be loader-bound on every fourth step. Look at the gaps, not the mean.

Which fixes belong to which stall?

Dataloader fixes live on the host path: more workers if the CPU is the wall, faster decode if you are unpacking images or tokens on the fly, a cache of ready batches, and fewer random small files per step. Adding GPUs to a loader-bound job multiplies idle devices.

Compute fixes live on the device path: a better kernel, a different batch shape, fewer unneeded synchronizes, or a collective that is not waiting on one slow rank. Prefetch will not shrink a 400 ms matmul. AI storage architecture matters when the loader is waiting on the fabric. It is irrelevant to a compute stall that already has the batch in HBM.

How do you run a one-axis test?

Feed a cached in-memory batch and disable augmentations. If step time barely moves, you were compute-bound. If step time collapses, you were loader-bound. Reverse the test: keep the real loader and replace the model with a tiny sleep kernel. If step time barely moves, the loader is the show.

OneSource Cloud exclusive hosts, including Texas / Richardson, remove a noisy neighbor who can make either stall worse. OnePlus Platform, OneSource Cloud’s AI orchestration platform, can keep the job on a dedicated project. Neither diagnosis is optional. The hall will not tell you which stall you bought.

FAQ

Is low GPU utilization always a dataloader problem?

No. Collectives, pipeline bubbles, and waiting on another rank also idle SMs. If the loader queue is full and the device is still idle, look at communication or a blocked host callback, not at worker count first.

Will more dataloader workers always help?

Only until you saturate CPU, memory bandwidth, or the storage QPS those workers generate. Extra workers can make a metadata storm worse. Raise them after you see CPU headroom and a still-empty prefetch queue.

Is this the same as storage-bound training?

Storage-bound is one cause of a dataloader stall. A stall can also be Python decode, a locked mutex, or a single worker on a huge shuffle. Name the stall, then name the medium.

Can a compute stall look like a loader stall in logs?

Yes, if you only print step time. A profiler that shows kernel spans versus host wait is the difference between buying disks and buying a better kernel. Do not buy either from a spreadsheet of averages.

Summary

A dataloader stall is a wait for the batch. A compute stall is work after the batch arrived. Trace the gap, run one-axis tests, and apply the lever that matches the pool of time you actually lost.

If the job deserves exclusive GPUs so the trace is not mixed with a neighbor, review OneSource Cloud private AI infrastructure and the home page, and keep the diagnosis honest.