Checkpoint I/O Bottlenecks in Multi-Node Training Storage

A checkpoint I/O bottleneck in multi-node training is a coordinated write burst that stalls every rank while model state hits storage, so GPUs sit allocated until the slowest writer finishes. It is not a single-disk “save is slow” ticket. It is N ranks times the checkpoint size, often on the same filesystem the loader is using.

Teams notice it as a periodic step-time cliff, NCCL waits, or a filesystem that looks fine between saves. The rest of this article is how the burst forms and how to keep it off the critical path without inventing benchmark numbers.

Why all ranks hurt at once

Distributed frameworks differ in how they shard state. The storage symptom is the same: a burst. Size the filesystem for that burst, not for average MB/s across an hour. Average MB/s is how NFS looks healthy until minute 47.

Design the burst, do not hope

Estimate checkpoint bytes from parameters, optimizer state, and whatever extra you dump. Multiply by how often you save. That is a throughput and metadata budget. If the store cannot take the burst in a fraction of a step, GPUs will idle. Local NVMe staging plus a background flush can hide the burst from the step. It cannot hide a flush that never catches up.

Residency still applies. Checkpoint copies are model weights and sometimes data shards. Do not flush them to a public bucket in another region because it was faster. Fast and wrong is a second incident.

Cluster and storage as one design

Multi-node GPUs without a checkpoint path are half a cluster. Exclusive nodes make the stall more expensive. Parallel filesystems, object stores with enough concurrency, and a network that is not also carrying all-reduce at save time belong in the private AI design.

OneSource Cloud’s AI storage architecture is meant to sit next to private AI infrastructure so checkpoint bursts have a volume. AI networking still matters when saves and collectives share a fabric. OnePlus, OneSource Cloud’s AI orchestration platform, should not start a gang job on a partition whose store cannot finish a save. Managed operations watch save duration as a first-class metric, not as a comment in a training log.

FAQ

Why do multi-node checkpoints idle every GPU?

Because ranks synchronize around the save. The slowest writer gates the step. If that writer is stuck on storage, every allocated GPU waits. It looks like a hang. It is often I/O. Profiling the save separately from compute is how you stop blaming NCCL first.

Should checkpoints go to object storage or a parallel filesystem?

Use a path that can take the burst and meet residency. Parallel filesystems often win on many large shards. Object storage can work with enough concurrency and large objects. Tiny-object dumps to a single prefix will stall either. Measure your actual shard layout. Do not copy a blog’s default.

Does async checkpointing remove the bottleneck?

It removes the stall from the step if the background flush keeps up and memory can hold the copy. It moves the bottleneck to a queue. If the queue grows forever, you will OOM or skip saves. Bound the queue and still size storage for the flush.

How often should we checkpoint?

Often enough that a preemption or node fail does not erase a week, rarely enough that save time is a small fraction of the step. The interval is a restart-cost decision. The I/O path is what makes a reasonable interval possible. Slow storage forces ugly intervals.

Can we checkpoint to the same volume we train from?

You can, and you will collide. Loaders and saves will fight for metadata and bandwidth. Split the volumes when gang size is large. Home directories are the worst checkpoint target. They were not sized for N ranks dumping optimizer state.

Summary

Multi-node checkpoint I/O is a burst that can idle every GPU. Shard large objects, split the volume, and size for the peak. If exclusive GPUs and training storage must be designed together, use OneSource Cloud AI storage with private AI infrastructure rather than dumping state onto the dataset NAS.