Why You Shouldn't Checkpoint to Local NVMe

When training large language models across distributed GPU clusters, checkpointing frequency represents a direct trade-off between compute efficiency and catastrophe insurance. If an engineer saves checkpoints infrequently to avoid pausing training, a hardware failure can discard days of compute progress. To minimize write latency, developers often write checkpoints directly to local server NVMe solid-state drives. While local NVMe offers blindingly fast sequential write speeds, relying on local instance disks for persistent training state is a dangerous anti-pattern. In multi-node distributed training, local checkpointing introduces single-point failure modes that can render distributed training runs completely unrecoverable.

The Structural Vulnerability of Local NVMe for Checkpoints

Local NVMe storage binds checkpoint data to the physical chassis of a single server; in distributed training involving dozens of nodes, this creates isolated data islands that cannot be accessed by replacement nodes when hardware faults occur.

Local NVMe drives are physically tied to the PCIe bus of an individual server chassis. In standard cloud environments, instance storage is ephemeral: when an instance terminates, experiences a hypervisor panic, or undergoes hardware maintenance, the local drive is wiped or inaccessible. Even on dedicated bare-metal servers, local storage creates an isolated data island.

Distributed deep learning frameworks (such as PyTorch FSDP, DeepSpeed ZeRO-3, or Megatron-LM) do not write a single consolidated file; they shard optimizer states and model weights across all participant ranks. If a 64-node cluster writes its state locally, 64 separate node-local drives hold fractional shards of the training state. The moment a single node experiences a hardware failure, its corresponding shard is trapped behind a powered-down motherboard, leaving the remaining 63 nodes with an incomplete, unusable checkpoint.

Node Failure Scenarios and Unrecoverable Training State

Distributed frameworks shard optimizer states and model weights across all ranks; if a node crashes and fails to reboot, its local shard is trapped, forcing the entire cluster to restart from an older shared checkpoint or abandon the training run completely.

In large-scale AI infrastructure, hardware failures are statistical certainties rather than rare anomalies. Across a cluster of 512 or 1,024 GPUs operating continuously at maximum power draw and thermal load, Mean Time Between Failures (MTBF) is measured in tens of hours. Common node failure scenarios that expose local checkpointing include:

• Unrecoverable Kernel Panics & PCIe Bus Faults: An uncorrectable PCIe bus error or memory fault forces a hard node reset. The cluster orchestrator immediately cordons the damaged node and provisions a healthy replacement. However, the replacement node cannot access the local NVMe drive of the cordoned server.
• Local NVMe Controller Hangs: Intensive checkpointing generates massive burst writes that saturate drive controllers and wear leveling logic. A controller timeout or drive lockup during a checkpoint save corrupts the local shard, invalidating the entire multi-node milestone.
• The Deadlock at Cluster Restart: When the distributed job attempts to resume from the latest checkpoint, the initialization barrier stalls waiting for Rank 48 to present its shard. Because Rank 48 is dead and its disk is local, the job fails. The entire cluster is forced to roll back to a distant shared snapshot—or restart the training run from scratch.

Parallel Shared Tiers: NVMe-oF and GPUDirect Storage

By implementing GPUDirect Storage (GDS) over NVMe-oF (NVMe over Fabrics), GPU memory communicates directly with parallel storage controllers over high-speed RoCEv2 fabrics, eliminating CPU memory bounce buffers and delivering multi-gigabyte-per-second write speeds without node lock-in.

To eliminate node-level storage isolation without sacrificing write performance, modern AI datacenters decouple compute from persistent state by deploying parallel shared storage over NVMe over Fabrics (NVMe-oF) with GPUDirect Storage (GDS).

In traditional network storage architectures (like standard NFS), writing a checkpoint requires copying data from GPU High-Bandwidth Memory (HBM) over the PCIe bus into host CPU system memory (RAM), buffering it through the Linux kernel page cache, and sending it through the host networking stack to a remote filesystem. This CPU bounce-buffer pipeline saturates host memory buses and introduces severe I/O stalls.

With GPUDirect Storage (GDS), the GPU DMA engine writes checkpoint buffers directly across the PCIe bus and high-speed RoCEv2 network adapters straight to remote NVMe-oF storage controllers, completely bypassing CPU memory and OS kernel bottlenecks:

Architectural Decision Matrix: AI Storage Architectures for LLMs

On OneSource Cloud's high-performance AI storage architecture, tiered NVMe-oF fabrics provide multi-gigabyte-per-second parallel write throughput directly accessible to every GPU node. When a node experiences a hardware fault, a replacement node provisions immediately and mounts the exact same shared checkpoint volume, resuming training within minutes.

Operational Boundaries: When Local NVMe Is Appropriate vs Forbidden

Local NVMe should be strictly reserved for read-only immutable dataset caching, local preprocessing caches, and ephemeral scratch logs; all stateful artifacts (model checkpoints, optimizer states, evaluation weights) must write to shared parallel storage.

Recognizing the risks of local NVMe does not mean local solid-state storage has no role in AI infrastructure. Rather, architectural best practice strictly delineates between ephemeral read caching and persistent state storage:

The operational boundary is definitive: never permit stateful checkpoint artifacts to reside exclusively on local instance storage. Local NVMe is a high-speed cache, while parallel shared fabrics provide the resilience required for enterprise AI training.

FAQ

Can local NVMe checkpoints be salvaged after a hardware kernel panic?

Generally no. When a physical server suffers a hardware fault or OS kernel panic, its local drives are disconnected from the cluster; without shared network access, remaining ranks cannot retrieve trapped shards.

How does OneSource deliver high-throughput checkpointing without local NVMe risks?

OneSource integrates GPUDirect Storage over NVMe-oF parallel storage tiers, writing checkpoints directly from GPU VRAM to shared storage at tens of gigabytes per second while eliminating node-level data isolation.