Set the Checkpoint Interval from Failures, Not from Habit

Most training jobs checkpoint because someone once picked 30 minutes and the cron still says 30 minutes. That number is unrelated to how long the save takes and unrelated to how often the job is interrupted. Write those two down and the interval falls out of a square root. Pick them badly and you either stop the GPUs to save a file you did not need, or you throw away an hour of work because the last good file is old. A second mistake sits beside the interval: saving the weights and calling it a resume. It is not.

The Interval Is a Square Root, Not a Cron Entry

Young's approximation, the first-order answer people still use because the next-order answers need data nobody has, puts the checkpoint interval near the square root of 2 × C × M. C is how long a save stalls the step. M is the mean time between interruptions of this job, counted in the same units. Interruptions include preemptions, node drains, and NCCL deaths, not only a GPU that threw an Xid.

An illustration, not a measurement of any cloud: a save that stalls for 2 minutes and a job that dies about every 6 hours. 2 × 2 × 360 is 1,440; the square root is about 38 minutes. Each death then throws away about half an interval of work, plus whatever the restart itself costs. A team that "checkpoints hourly" on that job is leaving goodput on the floor. A team that checkpoints every five minutes is spending the floor on saves. Neither looked at C or M.

Storage changes C. It does not pick the interval. A checkpoint tier that finishes a shard write without parking every rank is a smaller C, and then you recompute the square root rather than keeping the old cron. OneSource Cloud puts that tier on NVMe-oF next to the dedicated GPUs, so the save does not cross a shared multi-tenant pipe another job can fill. The arithmetic stays yours.

A Weight File Is Not a Resume

A checkpoint that can continue the same run, rather than a nearby run, holds more than parameters.

1. Model parameters, sharded the way the parallelism actually shards them.
2. Optimizer state. For Adam that means the moment buffers. Drop them and the loss jumps on resume because the optimizer has amnesia, which people then misread as a data bug.
3. Scheduler state and the global step.
4. Data-loader position: the sample or shard offset. Databricks' 2026 write-up of its training runtime is blunt about this one. Save the model and the step, forget the loader, and a restart repeats examples and skips others. Do it across enough restarts and the data distribution drifts with no exception thrown.
5. RNG state for dropout, shuffle, and augmentation, if you care that the resumed order matches.

The check after a restore

Load the file in a staging run and confirm the step, the optimizer moments, and the next sample id. A loss value that merely "looks fine" does not prove the loader was saved. Keep more than one complete checkpoint. A file written after silent corruption, or a file whose last shard never landed, should not be the only one you can roll back to. Async save is allowed to overlap the next step; it is not allowed to mark the checkpoint done when the bytes have only reached a local buffer.

Cut C and the Restart, Then Recompute

Buying more frequent checkpoints is the expensive way to buy goodput. Two cheaper levers show up in the same formula.

• Make C smaller. Asynchronous save, so the step pays a short stage and the write finishes behind it, cuts the stall. The same illustrative cluster, taken from a hardware-fault notebook rather than from a vendor benchmark, dropped a 2-minute synchronous stall to about 20 seconds and the sensible interval fell with it. Vendor quotes of "58× faster" belong to the configuration in the quote. Use them as a reason to measure your own C, not as your C.
• Make the restart smaller. A warm spare, an image that is already pulled, and a node that is not returned to the pool until it passes a check, cut the minutes after the failure. Those minutes are not in Young's formula and they dominate once C is small.

Recompute after either change. An interval chosen for synchronous NFS saves is the wrong interval on local NVMe, in both directions: you can now afford to save more often, and you should, because the work you lose is still half an interval. Write the new number next to the measured C and M. When M moves — a bad week of node drains, a driver update — the number moves. The cron does not get a vote.

Questions the formula does not answer by itself

What if I do not know my mean time between interruptions?

Use the last month of this job's actual deaths: preemptions, drains, collective timeouts, and GPU faults, averaged. Exclude kills you caused on purpose. That sample is a better M than a cluster-wide MTBF slide, because your job may be the one that dies on a single slow rank. Recompute when the month looks different from the month you used.

Does OneSource Cloud pick the checkpoint interval for a training job?

No. OneSource Cloud's NVMe-oF tier can shorten the write, which lowers C and therefore changes the interval you should set. The interval is still the square root of 2 × C × M from your job's own failure rate, and the file still has to contain optimizer state and the data-loader position or the resume is a different run.