How to Build Private AI Infrastructure From Sizing to Operations

NoraLin 9 2026-10-16 03:59:46 Edit

Private AI infrastructure is a dedicated compute environment that provides an organization with exclusive GPU resources, isolated networking, and full operational control for AI training and inference workloads. Building one is a project with five decision gates: workload sign-off, design review, facility readiness, acceptance testing, and production readiness. This guide walks through each gate using a 32-GPU reference design, so the decisions, quantities, and tests are concrete rather than abstract.

Before You Start: Is Building Justified?

Dedicated infrastructure pays off when demand is steady and control matters. It is usually premature when workloads are still experimental. Three checks help decide:

Check Signal that building is justified Signal to keep renting
GPU demand Rented GPUs are busy most hours of most days Usage comes in short bursts with long idle gaps
Data constraints Regulated, proprietary, or contract-restricted data in training or inference Public or synthetic data only
Access problems Projects delayed by quotas or unavailable multi-node capacity Capacity has been available when needed

If two of the three point toward building, the steps below apply. If not, rent while workloads stabilize and revisit in six months.

Gate 1: Workload Sign-Off

Write down the workloads the infrastructure must serve for the next 12 to 24 months and get the ML and product owners to sign them. Every later decision traces back to this list.

Workload Fields to capture
Training and fine-tuning Model sizes, training method (LoRA, QLoRA, full), tokens per run, runs per month, deadline per run
Real-time inference Models served, peak requests per second, context length, latency target, availability target
Batch inference Daily volume and completion deadline
RAG and embeddings Corpus size, refresh frequency, access-control model
Development Number of concurrent notebook users and typical GPU need per user

Two calculations turn this list into hardware. GPU memory per model instance (weights plus KV cache for inference, about 16 bytes per parameter for full training with Adam) sets the minimum GPUs per job. GPU-hours per month (6 × parameters × tokens for training, divided by measured throughput) sets the total GPU count.

Gate 2: Design Review With a Reference Design

A reference design gives the team something concrete to challenge. The example below is a 32-GPU cluster sized for regular full fine-tuning of models up to about 70B parameters, plus inference and development work. For a broader view of the layers involved, see the private AI infrastructure reference architecture.

Component Example specification Why it is sized this way
GPU servers 4 × 8-GPU H100-class servers (32 GPUs, 80 GB each, 2.56 TB total HBM) Full fine-tuning of a 70B model needs about 1.1 TB for weights, gradients, and optimizer state, plus activations
Compute fabric One 400 Gb/s port per GPU (32 ports) on a non-blocking InfiniBand or RoCE v2 fabric Gradient and parameter exchange between servers; a single 64-port 400 Gb/s switch covers up to 8 servers
Front-end and storage network Two high-speed Ethernet ports per server on a separate network Keeps checkpoint and user traffic off the compute fabric
Performance storage Parallel file system or NVMe-over-Fabrics tier with roughly 20 GB/s aggregate write Writes a 1 TB checkpoint in under a minute
Capacity storage Object storage for raw data, datasets, and archived checkpoints Lower cost per terabyte for data that is not on the hot path
Control plane Three management nodes for Kubernetes or Slurm control services Avoids a single point of failure in scheduling
Out-of-band network Separate network for BMC access Remote power control and recovery when servers hang
Software OS, GPU drivers, CUDA, NVIDIA GPU Operator or equivalent, DCGM, scheduler, monitoring, identity integration Makes hardware usable and observable

Questions to settle in the design review:

  1. Will the compute fabric need to grow beyond one switch? If the plan exceeds 8 servers, design a leaf-spine fabric now.
  2. Which workloads share GPUs through MIG partitions, and which need whole GPUs?
  3. Do inference endpoints run on the same servers as training, or on a separate pool?
  4. Where do datasets, checkpoints, model weights, vector indexes, and logs live, and who can access each?

Gate 3: Facility Readiness

GPU servers are far denser than typical enterprise servers. A DGX H100-class system is rated at 10.2 kW maximum input power in an 8U chassis, so the four servers above draw up to about 41 kW. Add switches, storage, and management nodes from their datasheets.

Facility item Reference design requirement What to check
Power About 41 kW for GPU servers at maximum, plus networking and storage Rack power distribution, circuit capacity, redundant feeds
Cooling Roughly 139,000 BTU per hour, about 11.6 tons, for the GPU servers alone Air cooling capacity per rack, need for rear-door or liquid cooling
Space and weight Four 8U servers at up to about 130 kg each Rack depth and floor loading
Connectivity Uplinks to corporate network and, if needed, cloud Bandwidth and security of the connection

Many corporate server rooms were built for racks well under 10 kW. If the facility cannot supply and cool the design, choose between upgrading it, colocation, or a hosted dedicated environment before ordering hardware. OneSource Cloud's Private AI Infrastructure offers dedicated hosted capacity in Richardson, Texas, which removes this gate for teams without a suitable facility.

Gate 4: Acceptance Testing

Acceptance testing is where self-built clusters most often reveal problems, such as a miswired fabric port or an undersized storage tier, that would otherwise surface months later as slow training. Define pass criteria before hardware arrives.

Test Tool or method What it proves
GPU health NVIDIA DCGM diagnostics at an extended level on every GPU No faulty GPUs, memory, or PCIe links
Burn-in Sustained GPU and memory load for 24 to 72 hours Early-life component failures appear before users arrive
Point-to-point fabric RDMA bandwidth and latency tests between every server pair Every link runs at rated speed
Collective communication NCCL tests (all-reduce, all-gather) across all GPUs Bus bandwidth matches the vendor reference for the same topology
Storage fio or vendor benchmarks for parallel read and checkpoint-size writes Data loading and checkpoint targets are met
Representative workload A real fine-tuning recipe at full cluster scale Tokens per second and scaling efficiency meet design targets
Failure handling Pull a node, kill a job, fail a management node Scheduler, monitoring, and restart behave as designed

Record baseline results. They become the reference for detecting performance drift during operations.

Gate 5: Production Readiness

Before opening the cluster to all teams, confirm that the people and process side is ready:

  • Quotas and priorities agreed with each team, and configured in the scheduler
  • Single sign-on, role-based access, and audit logging in place for clusters, data, and model registries
  • Approved container images and notebook templates published
  • Monitoring with alerts for GPU errors, node health, fabric errors, and storage latency
  • Runbooks for common incidents and an on-call rotation with defined response times
  • Usage reporting by team for chargeback or showback
  • A pilot team that has run real work for at least a few weeks

The OnePlus Platform, OneSource Cloud's AI orchestration platform, provides the scheduling, quotas, Kubernetes lifecycle management, developer workspaces, and usage reporting needed at this gate. See the OnePlus Platform.

What Determines the Timeline

The calendar time of a build depends mostly on a few long-lead items, which should start in parallel:

Work stream Typical dependency Can run in parallel with
GPU server procurement Vendor allocation and lead time Facility work, software design
Network and storage procurement Final design GPU procurement
Facility upgrades Power and cooling engineering, permits Procurement
Software platform build Design review Procurement and facility work
Identity and security integration Security team availability Software platform build
Acceptance testing All hardware installed Pilot team onboarding preparation

Hardware lead times change with market conditions, so confirm them with vendors at the start. Facility upgrades are the item most likely to add months, which is why many teams choose colocation or hosted capacity even when they want dedicated hardware.

The Team You Need to Run It

Role Responsibilities In-house or provider
GPU infrastructure engineer Servers, drivers, firmware, hardware failures Either
HPC network engineer Compute fabric, RDMA tuning, fabric monitoring Often provider; skills are scarce
Storage engineer Parallel file system or NVMe-oF tier, data paths Either
Platform engineer Kubernetes or Slurm, scheduler policy, images Either
MLOps engineer Training and serving pipelines, model registry Usually in-house
Security engineer Identity, segmentation, logging, compliance mapping Shared
On-call coverage 24/7 response Requires several people or a provider

Covering 24/7 operations in-house typically requires a rotation of several engineers with GPU, network, and storage skills. Organizations that want to keep ML work in-house and hand over the infrastructure layer can use Managed AI Infrastructure, which includes 24/7 operations, monthly SLA reporting, incident response, MIG configuration, and cost tuning. For a breakdown of what managed operations costs cover, see managed AI infrastructure cost line items.

Build In-House vs Provider-Built and Managed

Factor Build and run in-house Provider-designed, deployed, and managed
Control Full High, with dedicated hardware and defined access
Facility Yours to prepare Provider facility or yours
Design risk Found during acceptance or operation Proven designs, shared accountability
Staffing Full infrastructure team plus on-call Customer focuses on models and data
Cost profile Capital expense plus salaries Predictable contract-based spend

FAQ

How many GPUs should a first private AI cluster have?

Start from workloads. A team doing LoRA fine-tunes and inference may need one or two 8-GPU servers. Regular full fine-tuning of 70B models points to around 32 GPUs, because model and optimizer state alone need about 1.1 TB of GPU memory.

What acceptance tests should a GPU cluster pass?

At minimum: DCGM diagnostics on every GPU, a burn-in period, RDMA bandwidth tests between every server pair, NCCL collective tests across all GPUs, storage benchmarks at checkpoint scale, a representative training run, and failure-handling tests.

How much power does a private GPU cluster need?

Use datasheet maximums. An 8-GPU H100-class server is rated at up to 10.2 kW, so four servers need about 41 kW before networking and storage. Cooling needs are about 3,412 BTU per hour per kW.

Can we build private AI infrastructure without our own data center?

Yes. Colocation lets you own hardware in a provider's facility. Hosted dedicated infrastructure goes further, with the provider supplying facility, hardware, and operations while the environment stays single-tenant.

What is the hardest part of running private AI infrastructure?

Usually operations, not installation. Keeping drivers, firmware, fabric, storage, and scheduler healthy and performing at baseline requires specialized skills and 24/7 coverage.

Summary

Building private AI infrastructure works best as a gated project. Sign off workloads, then review a concrete reference design with quantities. Confirm the facility can power and cool it, define acceptance tests with pass criteria, and prove production readiness with a pilot team. The 32-GPU example shows how workload math drives every number, from GPU count to checkpoint storage throughput to cooling tonnage. Teams that want dedicated control without a facility project or an in-house operations rotation can have the same design delivered and operated by a provider.

To review a reference design against your workloads, start with OneSource Cloud's Private AI Infrastructure.

Tags:
Previous: Automated ML Deployment: Pipeline Design for Enterprise AI
Related Articles
新模块