Capacity Planning for Training vs Inference Methods

Capacity planning for AI training sizes for throughput — how fast a run completes — while capacity planning for inference sizes for latency and concurrency — how many simultaneous requests meet their latency targets — and applying the same method to both is why clusters are frequently the wrong size for the workload they actually serve. Training and inference have different demand shapes, different utilization targets, and different consequences of being undersized. Their capacity must be planned separately, even when the infrastructure is shared.

For teams deploying both training and inference, capacity planning is a split discipline. A cluster sized for training throughput will serve inference poorly, because inference needs headroom for latency peaks that training does not. A cluster sized for inference concurrency will train slowly, because training needs sustained throughput at scale that inference headroom wastes. The two workloads share GPUs but demand different sizing logic, and conflating them is the most common reason AI infrastructure is either starved or overbuilt.

This guide explains the different planning methods for training and inference, where the assumptions differ, and how to plan combined capacity. For the broader capacity sizing methodology, see how to size AI infrastructure capacity. For GPU-level throughput calculations, see how many GPUs for LLM training and H100 vs A100 for AI workloads.

Training Capacity Planning: Size for Throughput

Training capacity planning answers one question: how many GPUs, of what type, produce a training run of the target model and dataset size within the target wall-clock time. The method starts with the model size and dataset, computes the total training FLOPs, divides by the per-GPU achievable throughput (accounting for model FLOPs utilization, which is always well below peak), and adjusts for the parallelism strategy's communication overhead. The result is the GPU count and type that deliver the target training time. For the detailed method, see our LLM training GPU sizing guide.

Training capacity is batch-like: it absorbs whatever capacity is available and uses it at near-peak for the duration of the run. Utilization on a well-scheduled training cluster is high, because training jobs run continuously and pack GPUs efficiently. The planning challenge is not utilization but supply: training demand is lumpy (one job may need dozens of GPUs for days), and the cluster must either have enough capacity for the largest job or queue it until capacity frees.

Inference Capacity Planning: Size for Latency and Concurrency

Inference capacity planning answers a different question: how many GPUs, serving how many concurrent requests, meet the latency target at the expected concurrency, with enough headroom to absorb traffic peaks without breaking the target. Unlike training, which can queue, inference cannot — a request that arrives during a peak must be served within its latency target or the user experience degrades. This means inference needs headroom that training does not, consuming capacity that cannot be used for other work because it must be available when traffic spikes.

Inference utilization is therefore lower than training utilization by design. A latency-sensitive serving cluster runs well below peak utilization because it must reserve capacity for peaks. This is not waste; it is the cost of meeting latency targets under variable traffic. For how to monitor and manage inference latency, see token generation latency monitoring. For how batching raises throughput without breaking latency, see LLM inference batching.

Comparison of Planning Methods

Combined Capacity for Mixed Workloads

Many clusters run both training and inference, which requires planning that respects the different sizing methods. The patterns for combining them are partitioning (dedicated subsets for training and inference, each sized by its own method, with a hard boundary) and dynamic sharing (a pool of GPUs that a scheduler allocates by workload type and priority, with training filling the capacity inference does not need at any moment). For how orchestration handles this, see how AI orchestration works.

Partitioning is simpler but can waste capacity if the training partition is idle while inference is saturated, or vice versa. Dynamic sharing captures more utilization from the same hardware but requires a scheduler that understands the different requirements: inference is latency-sensitive and must not be starved or preempted; training can queue and tolerate preemption. For how to decide between managed and self-managed operations for shared clusters, see our decision framework.

Growth Planning for Both

Training and inference grow differently, which affects when and how to add capacity. Training demand grows with model and dataset size, which increase the GPUs per run and the number of concurrent runs. Inference demand grows with user traffic and model adoption, which increase concurrency and raise peak loads. The two growth curves rarely match, so capacity additions often address one workload type while the other absorbs spare capacity.

Plan growth by modeling each workload type's demand trajectory separately, then aggregating. A cluster that adds GPUs for inference growth will incidentally have spare training capacity when inference is below peak, which training can opportunistically use with the right scheduler. A cluster that adds GPUs for training growth will increase peak capacity that inference can use with headroom adjustment. Plan each workload's capacity independently and let the scheduler manage the sharing.

FAQ

How is capacity planning different for training and inference?

Training plans for throughput: the GPU count needed to complete a training run within a target time, with jobs queuable and utilization near-peak. Inference plans for latency and concurrency: the GPU count needed to serve peak concurrent requests within latency targets, with headroom for traffic spikes that training does not need. The methods are different and must be applied separately, even when the infrastructure is shared.

Can I use the same cluster for training and inference?

Yes, with the right planning and scheduling. Either partition dedicated subsets for each workload, sized by their respective methods, or use dynamic sharing where a scheduler allocates GPUs by workload type, with inference getting priority during traffic peaks and training consuming spare capacity. Dynamic sharing captures more utilization but requires a scheduler that preserves inference latency targets. See our orchestration guide.

What utilization should I target for inference?

Lower than training, by design. A latency-sensitive inference deployment must reserve capacity for traffic peaks, so its utilization runs below peak to stay within the latency budget. Planning for high utilization in inference typically means insufficient headroom, which translates to latency violations during peaks. Target a utilization level that keeps latency within target at peak concurrency, which varies by workload. See our latency monitoring guide.

How do I plan for inference traffic peaks?

Model expected peak concurrency, size GPU capacity to serve it within the latency target, and add headroom for the variability between expected and actual peaks. Unlike training, inference cannot queue requests during peaks without breaking latency commitments, so capacity must cover the peak. Autoscaling can help for predictable patterns, but the base capacity must handle the peak that autoscaling may not react to fast enough. For the economic tradeoffs of dedicated vs elastic capacity, see spot GPU vs dedicated capacity.

How does model size affect training vs inference capacity needs?

Larger models increase training's GPU count (through memory and compute) and inference's GPU count (through memory for weights and KV cache). But the proportion differs: training GPU count scales with model size more steeply because of optimizer state and activation memory, while inference GPU count scales more with concurrency and KV cache than raw model size. Plan each separately using the methods above, because the scaling behavior is different.

Summary

Capacity planning for training sizes for throughput; capacity planning for inference sizes for latency and concurrency. Training is lumpy, queuable, and targets high utilization. Inference is continuous, not queuable, and targets lower utilization by design to reserve for peaks. Combine them through partitioning or dynamic sharing with a scheduler that respects inference's latency requirements. Plan growth separately for each workload type, and let the scheduler manage the sharing. Applying the same method to both is why clusters are the wrong size for the workload they serve; applying both methods separately is what makes them fit. For the full capacity sizing methodology, see how to size AI infrastructure capacity.