Storage Architecture for LLM Training on GPU Clusters

GPU clusters stall on I/O when every training stream shares one mount. Dataset reads, checkpoint storms, trainer logs, and node scratch have different durability and bandwidth needs. Mixing them is how a failed node deletes the only recovery copy.

Training storage architecture is a tiered I/O design that assigns datasets, checkpoints, logs, and scratch to separate stores so multi-node trainers stay fed without treating ephemeral disks as recovery.

Name the streams first: dataset (hot and lake), checkpoints, logs, and scratch. Then place parallel filesystem, object, and local NVMe as roles. GPUDirect Storage can shorten a supported DMA hop. It is a path on that design, not the design. This page is training I/O, not RAG storage and not a hosting-control catalog.

Which I/O streams must the architecture name first?

Do not draw filesystems until each stream has an owner, a durability rule, and a stall budget. A single “training share” hides those rules until the first node death.

The lake is the system of record. The hot tier is what ranks open during an epoch. Checkpoints must survive a node kill. Logs explain a failed step. Scratch may vanish with the node.

Architectural Decision Matrix: AI Storage Architectures for LLMs

AI storage architecture should state that stream table before anyone mounts a path. OneSource Cloud treats the table as part of the cluster design, not as a mount option discovered after GPUs are already busy.

How should parallel filesystem, object, and NVMe be assigned?

These are roles, not a product shortlist. A parallel filesystem is the shared POSIX tier that stripes large sequential reads and coordinated checkpoint writes across storage servers. Object storage is the capacity and durability tier. Local NVMe is the per-node scratch tier.

Shard the dataset before you blame the filesystem. Millions of tiny sample files stall metadata even on a fast POSIX tier. Pack them into large sequential objects, then stage the working set onto the hot tier if the lake cannot meet the stall budget. Fine-tunes that fit on one node may skip the parallel tier. Multi-node pretraining usually cannot.

Checkpoint writes need a stall budget in seconds, not “as fast as possible.” Size for the burst when many ranks flush together, then copy a subset to object for residency and longer retention. A checkpoint that exists only on one node’s NVMe is scratch with a hopeful name.

Where does GPUDirect Storage fit in this architecture?

GPUDirect Storage, often shortened to GDS, is an NVIDIA I/O path that lets a GPU DMA file bytes toward device memory and skip a mandatory bounce buffer in CPU RAM on a supported mapping. The CPU still issues control. The path does not create a shared namespace or decide which stream is durable.

Enable it after the tiers exist and a profiler shows the host copy as the wait. Metadata storms, tiny files, or a congested storage network will not improve because the flag is on. A trainer that only issues POSIX reads may never use the path. Treat GDS as an optional hop, not as the architecture.

What networking and tenancy constraints bind the I/O design?

Remote dataset and checkpoint traffic rides a NIC and a fabric. If that fabric is already the limit for collectives, storage DMA cannot invent bandwidth. Write which rail carries storage and which rail carries training all-reduce. A single congested network makes every tier look slow.

High-performance AI networking is the constraint to name before you add another storage server. Private AI infrastructure keeps the trainers and the hot store in one control boundary. OneSource Cloud dedicated environments, including U.S. sites in Texas / Richardson, are one place to keep that boundary off quota swings. Research training programs still need the same stream map; a grant cycle does not make scratch a safe checkpoint target.

OneSource Cloud is a fit when the I/O table must sit on dedicated U.S. capacity, and a poor fit when a one-node fine-tune already fits on local disks plus a public lake.

What acceptance checks prove the training I/O design?

Accept measurements at planned job width, not a single-rank copy test. Average MB/s from a quiet afternoon is not a design.

1. Hot dataset read: at planned rank count, data-loader wait stays inside the written stall budget. GPUs idle on open or read fail the check.
2. Checkpoint storm: a coordinated flush finishes inside the stall budget. Partial files on one node do not count as success.
3. Node kill: scratch on the dead node is gone, and the job can resume from the shared checkpoint tier.
4. Object copy: a named checkpoint restores from object into the hot tier in the same residency set.
5. Log isolation: trainer logs do not contain dataset payloads or full checkpoint bytes, and they expire on the written retention clock.

Record the owner and last run date. Re-run after a layout, filesystem, or fabric change.

FAQ

What is storage architecture for LLM training, as opposed to a filesystem product?

It is the assignment of dataset, checkpoint, log, and scratch streams to stores with owners and stall budgets. A filesystem product is one role in that assignment. You can buy a fast POSIX tier and still fail if checkpoints live on scratch or if the lake is the only hot path. Name the streams first, then pick products.

When is object storage enough without a parallel filesystem?

When the job is narrow, shards are large and sequential, the stall budget tolerates object latency, and checkpoints are small or infrequent. Multi-node pretraining with frequent large flushes usually needs a shared POSIX hot tier plus an object copy. Direct-to-object training is a measured exception, not the default architecture.

Why must checkpoints leave local NVMe?

Local NVMe dies with the node. A preemption, a disk wipe, or a hardware fail loses the only recovery point if that was the checkpoint target. Keep the durable checkpoint on a shared tier and use local disks for scratch. An object copy then covers residency and longer retention.

Does enabling GPUDirect Storage replace the tier map?

No. GDS can remove a host bounce on a supported path. It does not decide which stream is durable, where replicas live, or how logs are retained. If epoch time is flat after you enable it, the wait was likely metadata or the network. Profile the wait before you treat the flag as architecture.

What cost drivers matter if list prices are unpublished?

Cost moves with hot-tier capacity, checkpoint retention, replica count, and the labor to run stall tests after every layout change. Object capacity is usually cheaper per terabyte than the POSIX hot tier. Compare those drivers in the I/O table. Unpublished fees are not a reason to put every stream on one mount.

How should research groups share a training store without mixing streams?

Give each group a quota on the hot tier, a lake prefix they cannot overwrite for others, and a scratch wipe policy that never touches checkpoints. Shared home directories are not a hot dataset tier. The stream map stays the same; only the quota and the project ID change.

How does OneSource AI Storage Architecture optimize LLM checkpointing and weight ingestion?

OneSource AI Storage Architecture pairs local PCIe Gen5 NVMe scratch storage on dedicated GPU nodes with a high-bandwidth distributed parallel storage fabric. By leveraging GPUDirect Storage (GDS) over RDMA networks, data transfers bypass host CPU memory bottlenecks entirely, allowing 70B+ parameter model weights to load in under ten seconds and preventing GPU compute cycles from stalling during heavy checkpoint write bursts.

Summary

Storage architecture for LLM training on GPU clusters assigns dataset, checkpoint, log, and scratch streams to matching tiers. Parallel filesystem, object, and local NVMe are roles. GPUDirect Storage is a path, not the map. If those tiers need dedicated U.S. capacity next to the trainers, review OneSource Cloud against the same I/O table.