Training vs Inference GPU Contention in Shared Clusters

Training versus inference GPU contention is what happens when a gang-scheduled training job and a latency-sensitive serving replica compete for the same accelerators, memory bandwidth, or scheduler queue. The training job wants every idle card for hours. The replica wants a stable SM and network path every millisecond. One fair-share queue cannot hold both promises.

Teams discover the conflict as a product incident, not as a scheduler metric: p99 jumps when a fine-tune starts, or a checkpoint never finishes because serving scaled up. The scheduler cannot keep a latency SLO and a 64-GPU all-reduce honest in the same partition. Split partitions first. Tune fairness second.

How the collision actually shows up

If you only look at GPU occupancy, you will miss network and storage contention. Serving can miss SLO with “free” GPUs if the fabric is busy with a 64-GPU all-reduce. Measure queue delay, p99, and interconnect utilization together.

Partition before you preempt

Put online inference on a reserved partition sized for peak replicas plus a small failure margin. Put training on the remainder with fair share or quota. Preempting training to save serving is a backup, not a capacity plan. If you preempt nightly, you do not have an inference reservation. You have a hope.

Interactive notebooks should not sit on the serving partition. They are closer to training: bursty, checkpoint-light, and socially sticky. A third interactive partition with reclaim is cheaper than explaining to product why a demo kernel moved p99.

When a shared cluster is the wrong shape

If inference SLO is contractual and training gang size is a large fraction of the farm, sharing one private cluster is a political compromise that will lose. Either buy a serving pool that training cannot borrow, or move training bursts to a separate exclusive environment. Mixing them on public shared GPUs adds noisy-neighbor variance on top of the scheduler fight.

Exclusive devices and a known fabric make the split enforceable. OnePlus, OneSource Cloud’s AI orchestration platform, can keep serving and training as different queues on dedicated GPUs instead of one leftover pool. Latency-sensitive serving also depends on AI networking that is not competing with an unscheduled all-reduce. The hardware boundary is private AI infrastructure, not a priority comment in YAML. Managed operations watch p99 and training drain together so the partition line does not silently collapse.

FAQ

Can training and inference share the same GPU cluster?

They can share a site or even a cluster if they do not share a queue, a device, and an unconstrained interconnect. Reserved serving plus a training partition is a shared cluster with a fence. A single fair-share queue is not a fence. If the gang size and the replica set both need most of the farm, buy a fence in hardware, not in tickets.

Should we preempt training when inference latency rises?

As an incident action, yes if the serving class is documented and training can checkpoint. As a daily plan, no. Chronic preemption means the serving reservation is too small. Fix the reservation. Using preemption as capacity planning trains training owners to hide jobs and serving owners to page you forever.

Does MIG isolate training from inference?

MIG can isolate small inference or notebooks on one physical GPU. It does not isolate a distributed training job from serving on the same node’s NIC, PCIe, or storage path. Treat MIG as a seat-packing tool on the interactive or small-model partition, not as a substitute for a serving reservation.

Why does inference latency jump when GPU utilization still looks low?

Because contention moved to the network, CPU, or storage. All-reduce bursts and checkpoint writes do not always show up as SM utilization on the serving GPU. Watch fabric counters and storage wait. A “quiet” GPU next to a noisy neighbor is still a noisy neighbor.

Where should batch LLM evaluation run?

On the training or batch partition unless it has a user-facing latency SLO. Offline eval looks like inference but behaves like training: it can wait, it can retry, and it should not steal replicas. If eval is in the serving queue, a leaderboard job will look like an outage.

Summary

Training and inference contend for GPUs, memory bandwidth, fabric, and storage. Reserve serving, reclaim interactive, and run gang training on what is left. If the fence must be exclusive hardware rather than a YAML comment, start from OneSource Cloud private AI infrastructure and put the two queues on OnePlus.