How to Size Reserved Inference vs Training Burst GPUs

Size reserved inference GPUs from the serving SLA, then size training-burst GPUs from the jobs you are willing to queue or preempt. A single shared pool that “usually has spare cards” will give training a good week and inference a bad afternoon. Traffic-spike sizing for one model is a different article. This one splits two workload classes on the same private cluster.

Reserved inference versus training burst is a capacity-partition decision that keeps serving GPUs out of the training scheduler’s borrow set. You make it when production APIs and experimental training share one hall and only one of them has a latency contract.

Platform and finance partners should write the split before the next large fine-tune lands. Changing it later means moving running replicas or pausing training. Dedicated hardware helps only if the scheduler respects the partition.

What are you sizing on each side?

Idle reserved inference is not waste until you have measured the peak and the failover replica. It is insurance. Training teams will call it waste. Show them the SLO math, not a utilization screenshot from 11 a.m.

How do you size the reserved inference partition?

Start from the live or forecast request mix: tokens in, tokens out, concurrency, and the tail percentile you sell. Convert that to replica count at the batching and cache settings you actually run. Add one replica for a node or replica failure if the SLA cannot wait for a cold start. Convert replicas to GPUs with the same SKU and tensor-parallel width you serve today.

Then add the silent loads: eval jobs that hit production weights, canaries, and embedding or guard models that share the serving path. If those land on the reservation without being counted, your “headroom” is fiction.

Do not size inference from average GPU utilization. A card at 40 percent can still miss TTFT when the batch queue spikes. Size from the SLO dashboard you already trust, or build that dashboard before you freeze the reservation.

How do you size the training burst?

List the jobs that must finish in the next planning window and the jobs that can wait. Burst capacity is the GPU-hours those must-finish jobs need, spread over the hours inference is not in its own peak if you share people or storage, not if you share GPUs. The point of the split is that training does not take inference cards.

Preemption belongs on the burst side. Checkpoint-friendly jobs can yield. Production serving should not. If a training job cannot checkpoint, it does not get to sit on a reservation “just this once.” Put that sentence in the policy.

Storage and network must match the burst. A sudden eight-node fine-tune on a thin checkpoint path will look like a GPU shortage. AI storage and AI networking reviews belong in the same sizing packet when the burst is multi-node.

How do you enforce and accept the split?

Use separate node pools or hard quotas, not a friendly email. Label the inference pool so the training queue cannot schedule there. If you have an orchestration layer, the reservation must be a scheduler object. OnePlus Platform, OneSource Cloud's AI orchestration platform, can hold per-team quotas so a research burst cannot consume a serving reservation. Quotas are not SLAs. They only make the split real.

Acceptance tests: fill the inference reservation with synthetic QPS and start a training burst that asks for “all free GPUs.” The burst must queue or land on the burst pool. Kill an inference replica and confirm the spare reserved capacity absorbs it. Then idle the inference pool at off-peak and confirm training still cannot borrow it without an approved break-glass path.

OneSource Cloud private AI infrastructure is a fit to evaluate when you want that split on dedicated U.S. GPUs rather than a public quota you do not control. It is the wrong tool if you only need a token API and have no training burst.

FAQ

Can training borrow reserved inference GPUs at night?

Only if a written policy says so, the borrow is time-boxed, and serving can reclaim cards faster than the SLA allows a miss. Most enterprise APIs still see overnight batch clients and global users. Default to no. If you allow nights, test reclaim, not only borrow.

Is this the same as reserved versus committed capacity with a vendor?

No. Reserved versus committed is a commercial construct about how you buy cards. This page is an operating partition on a cluster you already have or are designing. You can have a committed hall and still fail if inference and training share one schedulable pool.

How do we explain idle inference GPUs to finance?

Show the SLO, the peak mix, and the failover replica. Compare the cost of idle insurance with the cost of a latency incident. Utilization-only scorecards push teams to donate the reservation to training. Change the scorecard or the donation will win.

What if we have several inference apps?

Reserve by SLA class, not by every app name. Apps that share a failure domain can share a partition. Apps that cannot share noisy neighbors or data classes should not share a pool even if both are “inference.” Project isolation sits on top of this split.

Do we need the same GPU SKU on both sides?

Not always. Inference may want memory-bandwidth characteristics that training does not. Mixing SKUs is fine if the scheduler cannot place a training job on an inference SKU by accident. Document the allowed SKUs on each pool.

Summary

Sizing reserved inference versus training burst is an SLA-first partition: count serving replicas and failover, then give training a burst pool that can queue. Do not let average utilization donate the reservation. Enforce the split in the scheduler and prove it with a borrow test.

If that partition must live on dedicated U.S. GPUs with team-level quotas, review private AI infrastructure and OnePlus Platform as the environment and control plane, not as a substitute for the SLO math.