How to Mix Spot GPUs and Dedicated Capacity to Cut Inference Cost

Spot and dedicated GPU capacity are two ways to source inference compute: dedicated capacity is reserved and continuously available with predictable latency, while spot capacity is acquired at a lower price on an interruptible, best-effort basis. A hybrid strategy treats them as complements, using dedicated nodes for the baseline that must never fail and spot nodes for the peaks that can tolerate interruption.

This article explains which workloads belong on each tier and how to design a mix that controls cost without risking production latency.

Why a Single Capacity Model Falls Short

Production inference traffic is rarely flat. Average demand is lower than peak demand, and reserving enough dedicated capacity to absorb every spike leaves most nodes idle during quiet periods, raising cost. Relying purely on spot capacity, meanwhile, trades low price for unreliable availability and variable latency, which production services cannot accept.

A hybrid model keeps the predictable baseline dedicated and fills the difference between baseline and peak with the less expensive spot tier. The strategy works when each tier is assigned to workloads that match its availability profile.

Assign Dedicated Capacity to the Baseline

Dedicated capacity should carry the load that must respond within a latency budget at all times: interactive inference, live user traffic, and high-value production endpoints. Because these requests cannot be interrupted, the dedicated tier exists to hold them and to guarantee that even under peak demand the critical path stays within its service objectives.

Determine the dedicated baseline from the sustained and committed portion of traffic, not from the absolute maximum. Dedicated nodes fill this level; the unpredictable portion of demand is the candidate for spot.

Use Spot Capacity for the Elastic Surplus

Spot capacity fits load that can tolerate interruption or delayed completion: batch inference, pre-processing, background summarization, development and testing, and non-mission-critical bursts that can be retried. These workloads save money by riding the cheaper tier while the provider reclaims the node if demand returns.

Design spot handling so that an interruption is cheap. Keep state external, support retry and checkpointing, and route tolerant traffic so a reclaimed node does not stall the service. If a workload cannot accept a mid-request interruption, it does not belong on spot.

Design the Routing and Fallback Logic

The value of a hybrid model depends on routing. Route latency-sensitive requests to the dedicated tier and queue or deduplicated requests to the spot tier. When spot capacity is unavailable, decide whether to wait, shed the request, or fall back to dedicated capacity, and define the trade-off in advance.

1. Segment the workload: classify requests by latency tolerance and criticality before routing.
2. Reserve the baseline: size the dedicated tier to the committed, always-on portion of demand.
3. Add the elastic tier: attach spot capacity for the tolerant surplus between baseline and peak.
4. Define interruption policy: document how each workload reacts when a spot node is reclaimed.
5. Monitor the split: track utilization, cost, and the ratio of dedicated to spot usage over time.

OneSource Cloud AI orchestration platform can place dedicated inference capacity and quota controls under one pool and monitor usage across teams. Teams that want a predictable dedicated baseline without operating the scheduling layer themselves can evaluate private AI infrastructure with fixed capacity.

FAQ

What is the difference between spot and dedicated GPU capacity?

Dedicated capacity is reserved infrastructure that is continuously available with predictable latency and a stable cost. Spot capacity is inexpensive but interruptible, meaning the provider can reclaim the node at any time, so it is best-effort and unsuitable for latency-sensitive workloads. The difference is availability and predictability in exchange for price.

Which workloads should never run on spot capacity?

Any workload that cannot tolerate an interruption: interactive inference, live user requests, real-time production endpoints, and services with strict latency or reliability budgets. These belong on dedicated capacity. Batch processing, retryable background jobs, and tolerant burst traffic can safely use spot because a reclaimed node is a delay, not a failure.

How much can a hybrid GPU strategy save?

Savings depend on how much traffic is tolerant and how often the spot tier is available in the region. Shifting bursty, tolerant workloads to a cheaper tier can reduce waste from idle dedicated capacity, but the number varies by workload and region. Model the dedicated baseline, the elastic surplus, and the fallback cost to estimate real savings rather than assuming a fixed discount.

Do I need dedicated capacity for all inference traffic?

No. Dedicated capacity is required only for the portion that must stay available with predictable latency. The elastic surplus that can tolerate interruption, retry, or delayed completion can run on a cheaper, interruptible tier. The right split depends on your consumers, latency budget, and the cost of an interrupted request.

Summary

A hybrid GPU inference strategy uses dedicated capacity to guarantee the always-on baseline and spot capacity to absorb the tolerable surplus between baseline and peak. Each tier must be matched to the workload's availability tolerance, with routing and fallback logic defined in advance. This design controls cost without putting latency-sensitive production traffic at risk.