How to Size a GPU Cluster for AI Training

A GPU cluster for AI training is a group of GPU servers connected by a high-bandwidth, low-latency network and backed by fast shared storage, so that a single training job can be distributed across many accelerators. Sizing one takes four calculations: compute (how many GPU-hours the job needs), memory (whether the model and optimizer state fit), interconnect (whether GPUs can exchange gradients fast enough), and storage (whether data loading and checkpoints keep up). The worked examples below show how each calculation changes the answer.

The Four Inputs You Need Before Sizing

Every sizing exercise starts from the workload, not a hardware quote. Collect these four inputs for each training job you expect to run in the next 12 to 24 months:

Many enterprise teams assume they need a large cluster because "training" sounds expensive. As the examples show, the training method changes the answer more than model size does.

Step 1: Estimate Compute in GPU-Hours

For transformer models, a widely used approximation is that training costs about six floating-point operations per parameter per token:

Training FLOPs ≈ 6 × N × D

To convert FLOPs into GPU-hours, divide by the throughput a GPU actually sustains. An NVIDIA H100 SXM delivers about 989 TFLOPS of dense BF16 Tensor Core compute on its datasheet. Real training reaches only a fraction of that, measured as model FLOPs utilization (MFU). Meta reported 38–43% MFU for Llama 3 405B training on H100 clusters, which is a reasonable planning reference for a well-tuned job. The examples below assume 40%, or roughly 396 TFLOPS sustained per GPU.

The LoRA figure is an upper bound because LoRA skips most weight-gradient computation. The point stands either way: a typical enterprise fine-tune fits comfortably on a single 8-GPU server, while continued pre-training at 1,680 GPU-hours takes about 53 hours on 32 GPUs or about 26 hours on 64 GPUs, if scaling efficiency holds.

Before committing to a cluster size, run a short benchmark of the real job on candidate hardware and measure tokens per second. Measured throughput replaces the MFU assumption and is the single most useful number in the sizing exercise.

Step 2: Check Whether the Model Fits in GPU Memory

Compute tells you how long a job takes. Memory tells you the minimum number of GPUs required to run it at all, and it often forces a larger cluster than compute alone would suggest.

For full training with the Adam optimizer in mixed precision, a common estimate is about 16 bytes of GPU memory per parameter: 2 bytes for BF16 weights, 2 for gradients, and 12 for FP32 master weights and two optimizer moments. Activations come on top and depend on batch size and sequence length.

With 80 GB per H100, full fine-tuning of a 70B model needs at least 14 GPUs just to hold model and optimizer state, before activations. In practice that means sharding with FSDP or DeepSpeed ZeRO across four 8-GPU servers. Compare that with the compute estimate: 590 GPU-hours would take about 74 hours on one server, but the job cannot run on one server at all. Memory, not compute, is what makes this a multi-node job.

The same 70B model with QLoRA fits on a single 80 GB GPU. Choosing the training method is therefore the first sizing decision, and it should be made with the ML team before hardware is ordered.

Step 3: Size the Interconnect

Once a job spans servers, GPUs exchange gradients or sharded parameters at every step, and the network becomes part of the training loop.

Within an 8-GPU H100 server, GPUs communicate over NVLink at 900 GB/s per GPU. Between servers, a DGX H100-class system has eight ConnectX-7 adapters, one per GPU, each running at up to 400 Gb/s, which is 50 GB/s per GPU. That is an 18x drop in bandwidth the moment traffic leaves the server, which is why parallelism strategies keep the most communication-heavy work, such as tensor parallelism, inside a server.

A simple calculation shows why per-GPU network bandwidth matters. In pure data-parallel training of an 8B model, each step synchronizes about 16 GB of BF16 gradients. A ring all-reduce moves roughly twice that per GPU, about 32 GB.

Frameworks overlap communication with computation, so these times are not added directly to every step. But when communication time approaches computation time, adding nodes stops reducing training time. That is the pattern teams describe as "training speed flattened after four nodes."

InfiniBand or Ethernet

Both InfiniBand and RDMA over Converged Ethernet (RoCE v2) can deliver strong training performance. The protocol matters less than four design properties:

• One high-speed port per GPU on the compute fabric, rather than shared ports
• A non-blocking or rail-optimized topology, so all GPUs can communicate at once without oversubscribed uplinks
• A separate network for storage and management, so checkpoint writes do not compete with gradient traffic
• Validated collective performance, measured with NCCL tests across all nodes before handover

InfiniBand tends to need less tuning for lossless behavior. RoCE requires careful configuration of priority flow control and congestion management, but it fits organizations that already run large Ethernet fabrics. OneSource Cloud's AI Networking Services cover fabric design and validation for multi-node training.

Step 4: Size Storage for Data Loading and Checkpoints

Storage has two separate jobs in a training cluster, and each needs its own target.

Data Loading

Text training is rarely storage-bound because tokens are compact. Image, video, audio, and multimodal training is different: every step reads large files, and all nodes read at the same time. Local NVMe on each server works as a cache for datasets that fit, and DGX H100-class systems include about 30 TB of local NVMe for that purpose. Datasets larger than local cache need a shared parallel file system or high-throughput object storage, optionally with GPUDirect Storage to move data directly into GPU memory.

Checkpoints

Checkpoints are the storage cost teams most often underestimate. A full checkpoint of the 70B example, including FP32 master weights and optimizer state, is roughly 1 TB. The write speed determines how long training pauses:

At 5 GB/s with synchronous checkpointing every 30 minutes, roughly one-ninth of the cluster's time goes to waiting on storage. Asynchronous checkpointing reduces the pause, but the data still has to land before the next checkpoint starts. Size the storage tier for checkpoint bursts, not only for capacity. See AI Storage Architecture for tiered designs that separate fast checkpoint storage from capacity storage.

Step 5: Plan Power, Cooling, and Space

Training clusters are dense. A DGX H100 system is rated at 10.2 kW maximum input power in an 8U chassis, so a four-server cluster draws up to about 41 kW before counting switches and storage. Converting to cooling load at 3,412 BTU per hour per kW, that is roughly 139,000 BTU per hour, or about 11.6 tons of cooling.

Uptime Institute's global surveys have put average data center rack density in the single-digit kilowatts, with relatively few facilities supporting racks above 30 kW. A training cluster that fits physically into a corporate server room may still exceed its power and cooling capacity. Confirm power per rack, cooling method, and floor loading before choosing between on-premise, colocation, or hosted capacity.

Step 6: Plan for Utilization, Not Just Capacity

A 64-GPU cluster running at 35% utilization delivers the useful work of about 22 GPUs. Utilization is the number that determines the real cost per training run, and it is usually limited by things other than hardware.

Track GPU activity with NVIDIA DCGM metrics such as SM activity and Tensor Core activity, not only the basic "GPU utilization" counter, which reports a GPU as busy even when it runs inefficient kernels. The OnePlus Platform, OneSource Cloud's AI orchestration platform, adds GPU scheduling, quotas, and usage reporting on dedicated clusters. See the OnePlus Platform.

Putting It Together: Three Sizing Outcomes

These are starting points derived from the calculations above. A benchmark on the target hardware, with your data and framework, should confirm them before purchase or contract.

Rent, Build, or Dedicate

OneSource Cloud's AI GPU Cluster service designs, deploys, validates, and operates dedicated training clusters, with U.S.-based capacity in Richardson, Texas, and 24/7 operations through Managed AI Infrastructure.

FAQ

How many GPUs do I need to fine-tune a 70B LLM?

With QLoRA, one 80 GB GPU can hold a 70B model. With LoRA on BF16 weights, plan for at least two 80 GB GPUs for the weights alone. Full fine-tuning needs roughly 1.1 TB for model and optimizer state, so plan for around 32 H100-class GPUs across four servers with sharding.

How many GPU-hours does it take to fine-tune an LLM?

Use 6 × parameters × tokens to estimate FLOPs, then divide by sustained throughput. At 40% MFU on H100, a LoRA fine-tune of an 8B model on 200M tokens takes under 10 GPU-hours, while a full fine-tune of a 70B model on 2B tokens takes around 600.

Is InfiniBand required for multi-node training?

No. Well-designed RoCE v2 Ethernet fabrics also perform well. What matters most is one high-speed port per GPU, a non-blocking topology, a separate storage network, and validated NCCL performance.

Why does my training job get slower per GPU when I add nodes?

Communication time is growing faster than compute time. Common causes are oversubscribed network uplinks, too little bandwidth per GPU, misconfigured RDMA, or a parallelism strategy that sends tensor-parallel traffic across servers.

How much storage throughput does an LLM training cluster need?

Text data loading is usually light. Checkpoints set the requirement: divide the checkpoint size by the pause you can accept. A 1 TB checkpoint written in under a minute needs roughly 20 GB/s of aggregate write throughput.

Summary

Sizing a GPU cluster for AI training is four calculations. Compute (6 × N × D divided by sustained throughput) sets GPU-hours. Memory, at about 16 bytes per parameter for full training, often sets the minimum GPU count. Per-GPU network bandwidth decides whether added nodes speed up training. Checkpoint write throughput decides how much time the cluster spends waiting. The training method changes the answer most, so settle it with the ML team first, then confirm the sizing with a benchmark on target hardware.

To turn these estimates into a validated cluster design, explore OneSource Cloud's AI GPU Cluster service.