How GPU Reclaim and Preemption Work in AI Operations

GPU reclaim is a policy that takes unused burst capacity back so a waiting team can start, and GPU preemption is a policy that stops a running job so a higher class can have the cards now. Reclaim fights hoarding. Preemption fights priority inversion. If you only have one verb for both, people will treat every eviction as an insult.

Operations teams need a published sequence: who may burst, how long idle burst may last, what signal starts reclaim, whether the victim can checkpoint, and which classes are never preempted. Without that sequence, Slack becomes the scheduler.

Reclaim vs preemption vs quota

Fair share may decide who is entitled to reclaim. It should not silently kill production. Put serving in a class that reclaim cannot touch unless an incident commander says so.

How reclaim should work in practice

Burst is a loan. The loan needs a timer and a watchdog. Typical victims are Jupyter kernels that allocated a full GPU, debug pods left in Running, and departmental extras taken when the cluster was empty. Warn, then drain, then force. A fifteen-minute warning that nobody can see is not a warning.

Count reclaim against the same identity as quota. If a contractor notebook is billed to a shared lab account, you will reclaim the wrong human and the lab will disable the watchdog. Identity is the whole mechanism.

How preemption should work without losing a week of training

Preempt only jobs that can restart. That means checkpoint-friendly training, not a twelve-hour uncheckpointed compile, and not a serving replica that is the last healthy pod. Give a grace period that matches checkpoint time on your storage, not a generic thirty seconds copied from a web microservice.

Publish a class ladder. Example: interactive burst, then low-priority training, then default training, then production inference. Research “deadline Friday” is not a class. If you allow it, you will preempt the wrong victim every term.

Failure modes AI operations should expect

Checkpoint to a slow object store makes preemption look like data loss. Gang jobs that lose one GPU and keep the rest become zombies. Preempting the dataloader host but not the GPU workers leaves allocated devices with no trainer. Test the path with a cheap job before you enable it on the 70B run.

On a dedicated multi-team cluster, reclaim and preemption have to be platform features, not a runbook. OnePlus, OneSource Cloud’s AI orchestration platform, is the layer that binds identities, queues, and workspace lifetime on private GPUs. Checkpoint speed is an AI storage constraint as much as a scheduler one. Exclusive hardware via private AI infrastructure does not remove the need for reclaim; it makes the cards expensive enough that idle burst is visible. Managed operations keep the watchdog from dying after the first successful week.

FAQ

What is the difference between GPU reclaim and GPU preemption?

Reclaim takes back unused or borrowed capacity, usually idle interactive GPUs or burst extras, because someone entitled is waiting. Preemption stops a job that is still running because a higher class needs the devices now. One is about waste. The other is about priority. Using one ticket type for both trains people to fight every drain.

How long should the reclaim grace period be?

Long enough for a human to save a notebook or for a training job to write a checkpoint, short enough that hoarding is not rational. Interactive reclaim is often minutes. Training preemption should match your real checkpoint duration on the storage you actually use. Copying a microservice termination grace period will cut jobs mid-write.

Can we preempt a distributed training job on one GPU only?

Usually no. Gang jobs need all ranks. Preempt the whole allocation or none of it. Partial eviction leaves NCCL ranks hanging and the remaining GPUs allocated. If the scheduler cannot evict the gang cleanly, do not enable preemption on that partition.

Should production inference ever be reclaimed?

Not by the default idle watchdog. Serving headroom can look idle and still be required for tail latency. If you borrow serving GPUs for overnight training, reclaim must beat the incident clock and the model must be warmable in that window. Treat that as an explicit exception, not a fair-share side effect.

Does fair share require preemption?

No. Fair share can wait for jobs to finish. Preemption is optional acceleration for higher classes. Many research clusters run fair share without preemption and still reclaim idle notebooks. Add preemption only after checkpoint and gang eviction are proven.

Summary

Reclaim idle burst. Preempt only restartable lower-class work. Publish the ladder, the grace period, and the identities before the first eviction. If that control plane needs to sit on dedicated GPUs with real team accounts, look at OnePlus on OneSource Cloud and test reclaim on a cheap job before you aim it at the long training run.