How to Detect Stalled Training Runs on GPU Clusters

Quick Answer: Detect a stalled training run by watching whether steps still finish, loss still logs, and every rank still heartbeats. Do not trust SM% alone. A hung collective can keep cards busy while the job makes no progress.

A stalled training run is a job that still holds GPUs but has stopped completing optimizer steps or writing the agreed progress signals inside a watchdog interval. Operations kill or probe it so the reservation returns to a queue. Slow jobs are not stalls. Silent jobs are.

This page is a detection method. It is not a hardware Xid catalog and not a collective-library timeout deep dive. If the GPU fell off the bus, use the hardware path. If the cards are present and the step timer is dead, stay here.

Which signals mean progress, and which only mean occupancy?

Define the watchdog in step time, not in wall time alone. A step that usually takes 2 seconds and last finished 20 minutes ago is stalled. A step that usually takes 18 minutes on a huge batch is not stalled at 19 minutes. Store the expected interval with the job, or the page will fire on every new model.

How do you instrument the job so the cluster can see it?

Emit a progress record at the same place you commit an optimizer step. Include step number, timestamp, rank, and job ID. Write it to a place the platform can read without attaching a debugger. A notebook print is not instrumentation.

Require a heartbeat from every rank, not only rank 0. Rank 0 can still log while rank 11 is stuck in a fetch. The group is stalled. If you cannot get per-rank heartbeats, at least fail when the step interval exceeds the budget. Partial visibility is better than SM% theater.

Flush logs. Line-buffered loss that sits in a 64 MB buffer will look fine until the process dies. The watchdog should read the progress file, not the pretty dashboard that updates when someone tails a file.

How do you separate stall from slow I/O or a long compile?

Startup compile, first-batch download, and a planned evaluation epoch should set a different timer. Label those phases. A job in “compile” for three hours on a node that already compiled yesterday is stalled. A job in “compile” for eight minutes on a cold worker may be healthy.

I/O-bound steps still complete. They complete slowly and often show low SM%. That is a storage or dataloader ticket, not a stall kill. If steps still finish, widen the interval or fix the path with AI storage architecture. Killing a slow-but-live job destroys a checkpoint you could have waited for.

Collective hangs complete nothing. They often show high SM% and stable power. That is a stall. Capture a stack on one rank if you can do it without making the hang worse, then kill and resume from the last committed checkpoint. Do not sit overnight “to see if it comes back” on a 64-GPU reservation.

What should the platform do when the watchdog fires?

Page the job owner with the last step number and the interval. Do not page the whole company. If the owner does not ack, execute the documented action: collect a short artifact, then preempt. Write the action before the first fire. Debating in the ticket at 3 a.m. is how idle cost returns.

Resume from the last committed snapshot, not from an in-flight async flush. If you do not have a committed generation, you are restarting. That pain is what makes checkpoint catalogs worth staffing. Other pages cover async snapshots. Detection only needs to know the last good ID.

Keep training on exclusive nodes so a neighbor’s interactive kernel is not your hang. Private AI infrastructure does not prevent a deadlock. It removes one class of noisy neighbor. OneSource Cloud U.S. fleets, including Texas / Richardson, still need the same watchdog.

Which false pages should you tune out?

Do not alert on a single delayed step after a known eval. Do not alert because SM% dropped during a live dataloader prefetch that still finishes. Do not alert because a plot logger paused. Alert because the step contract broke.

OnePlus Platform, OneSource Cloud's AI orchestration platform, can surface the progress file next to the quota so on-call sees occupancy and step age together. The platform cannot invent a step counter the trainer never wrote. Instrument first.

Managed AI infrastructure can own the pager and the preempt. The training team still owns the definition of a step. Write that definition in the job spec.

FAQ

Is high GPU utilization proof the run is healthy?

No. Utilization proves the device is busy. Busy can mean useful kernels or a wait loop. Progress is a completed step. If you only graph SM%, you will praise a hang that is burning a reservation.

How is this different from hardware error handling?

Hardware pages when the device reports a fault or disappears. Stall pages when the device is still there and the trainer stops completing steps. You can have both. Run the hardware path when the device is gone. Run this path when the device is present and silent.

Should we auto-kill on the first missed interval?

Usually no. Use two intervals or a grace for labeled phases. Auto-kill is appropriate after a repeated miss with no heartbeat. The exact counts belong in your job class. A 2-GPU debug job and a 256-GPU pretrain should not share the same grace.

Can we detect stalls without changing trainer code?

You can watch log timestamps and process liveness, and you will get more false pages. A one-line progress write per step is cheaper than a week of ghost reservations. If you cannot change the trainer, wrap it and parse a contract you document.

Where do checkpoints fit the detection loop?

Detection decides the job is dead. Checkpoints decide how far you roll back. A perfect watchdog with no snapshot still restarts from zero. Build both. Detection without restore is just expensive alerting.

Summary

Stalled training is a broken step contract, not a low SM% chart. Instrument step completion and rank heartbeats, label long phases, and preempt when the watchdog fires. Occupancy metrics stay for capacity. They do not declare a run alive.

Run the jobs on exclusive GPUs so interactive leftovers are not in the story, and keep a committed snapshot to resume. OneSource Cloud can host that fleet and the pager under managed operations. Add the progress file before you add more cards. Start from the OneSource Cloud cluster operations conversation when the missing piece is a named environment rather than another dashboard.