Async Checkpointing for GPU Training Workloads

Quick Answer: Async checkpointing starts a snapshot and lets the next training steps run while the copy lands on storage. It is not a free checkpoint. If the flush is incomplete, the file is not a restore point.

Async checkpointing is a training snapshot method that overlaps most of the write with continued computation so GPUs spend fewer steps blocked on storage. Teams use it when synchronous checkpoints waste a noticeable fraction of the job. They still need a complete, versioned copy they can boot after a kill.

This page defines the method and the failure cases. It is not a residency checklist for checkpoint copies and not a parallel-filesystem sizing guide. Those pages assume you already know whether the write is allowed to overlap compute.

How does async differ from a blocking checkpoint?

Blocking is easier to reason about and slower on fat models. Async is faster on the happy path and stricter about generation IDs. If your trainer cannot tell a committed snapshot from a partial file, do not enable overlap. You will “resume” into corruption and blame the framework.

What must be true before you overlap the write?

Weights, optimizer state, and any extra tensors you need for resume must be captured as a consistent generation. Some stacks copy to host memory first, then flush. Some stage to local NVMe, then copy to shared storage. Name the stages. A job that writes shards from rank 0 through rank N without a barrier is not async. It is a race.

The file name or catalog must not be reused until the generation is committed. Readers resume from the last committed ID, never from “whatever is newest in the folder.” Newest-in-folder is how a kill during flush becomes a silent bad boot.

Storage must accept the burst without freezing the next iteration’s data path if you still read datasets from the same appliance. Put checkpoints on a path that AI storage architecture can isolate from the training read stream. Shared niceness is not a consistency protocol.

When does async checkpointing pay off?

It pays off when a blocking write is a material slice of step time and the copy engine can run without stealing the interconnect you need for the next all-reduce. Fat optimizer states on a slow NFS export are the usual pain. Small models on a fast parallel file system may not notice.

It does not pay off when you checkpoint every step, when host memory cannot hold the staging copy, or when you cannot staff a restore test. Overlap is an optimization. Restore is the product. Teams that never reboot from the async file have a write benchmark, not a checkpoint strategy.

Frequency still matters. Async does not mean “checkpoint constantly.” You still choose a step interval from loss-of-work versus I/O tax. Overlap only reduces the tax. It does not make an interval of one step free.

What breaks in multi-node jobs?

Ranks can finish their local stage at different times. If rank 7 is still flushing while rank 0 marks the generation committed, a kill leaves a franken-snapshot. The commit record belongs after every rank reports durable. That is a distributed commit, not a log line on the driver pod.

A later step can mutate buffers you thought you had already copied. The capture must be against a frozen view. If your library mutates in place, you need a copy or a guaranteed freeze. This is why “just thread the write” is not a plan.

Do not land shards only on local NVMe and call the job recoverable. Node death deletes the shard. Stage locally if you must, then replicate to durable shared storage before commit. Exclusive nodes on private AI infrastructure still die. OneSource Cloud can host the durable target in the same U.S. boundary as the GPUs, including Texas / Richardson, when the weights cannot leave that boundary.

How should operations watch it?

Alert on commit lag: steps since last committed generation. Alert on flush errors even if loss is still moving. A pretty loss curve with a dead copy engine means you will restart from an older generation than you think.

Practice restore on a scratch partition, not only on the lucky job that finished. Measure time-to-first-step after resume. If resume needs a human to pick among three half-written directories, the catalog is unfinished.

OnePlus Platform, OneSource Cloud's AI orchestration platform, can keep the training job and the copy traffic inside a quota so a checkpoint storm does not evict interactive work. Quota is not a commit protocol. The trainer still owns the generation ID.

FAQ

Is async checkpointing the same as gradient checkpointing?

No. Gradient checkpointing trades compute for activation memory during the backward pass. Async snapshotting trades a consistency protocol for less blocked I/O during a save. Confusing the two words in a design review will waste a week. Use “snapshot” in tickets if your team keeps mixing them.

Can we resume while a flush is still running?

Only from a previously committed generation. Resuming from an in-flight flush is how you load torn optimizer state. Wait for commit, or start from the prior ID. There is no safe “almost written” resume.

Does faster storage remove the need for async?

Sometimes. If a blocking write is already a small slice of step time, overlap adds protocol risk for little gain. Measure the blocked interval first. Buy overlap when the interval is large and restore tests pass. Do not enable it because a slide said async is modern.

What if the job is killed during the host staging copy?

The last committed generation remains the restore point. The in-flight generation should be discarded by the catalog. If your code leaves partial shards with the same name as a real generation, fix the naming before you enable overlap in production.

Who owns the snapshot path on a managed cluster?

The training owner owns the generation and the resume command. Platform owns the durable filesystem and the quota. Managed AI infrastructure can watch flush errors and capacity. It cannot invent a commit record the trainer never wrote.

Summary

Async checkpointing overlaps snapshot I/O with later training steps. The GPUs keep moving. Restore still requires a committed, complete generation. Blocking writes are slower and simpler. Overlap is an optimization you earn with a catalog and a restore drill.

Land committed copies on durable storage in the same data boundary as the job. OneSource Cloud can provide exclusive GPUs and that storage path. Enable overlap only after a scratch resume works. Review AI storage architecture when the current export cannot isolate checkpoint bursts from dataset reads.