Enterprise AI Infrastructure Requirements for Production AI

Enterprise AI infrastructure is the combination of GPU compute, storage, networking, orchestration software, security controls, and operations processes that an organization uses to train, fine-tune, and serve AI models in production. The hardware is the easy part to specify. What separates enterprise infrastructure from a pilot environment is that every layer has a measurable requirement and a named owner. This page sets out those requirements by layer and by stage, with the calculations teams need to turn a workload list into a specification.

Pilot, Production, and Scale Have Different Requirements

Most requirement documents fail because they describe a single target state. AI programs move through stages, and each stage changes what "enough" means.

The transition from pilot to production is where most programs stall. Usually the infrastructure has not been redesigned for committed capacity, governed data, and round-the-clock operations.

Layer 1: Compute Requirements

Compute requirements should come from workloads, expressed in two numbers: GPU memory needed per model instance and GPU-hours needed per month.

Worked Example: GPU Memory for LLM Inference

Serving a large language model needs memory for the model weights plus the key-value (KV) cache, which grows with the number of concurrent requests and their context length. The KV cache size per token is:

KV cache per token = 2 × layers × KV heads × head dimension × bytes per value

For Llama 3.1 70B (80 layers, 8 KV heads, head dimension 128) in BF16, that is 2 × 80 × 8 × 128 × 2 bytes, or about 0.33 MB per token.

Quantizing weights and KV cache to FP8 roughly halves both figures, bringing the 70B example to about 113 GB, which fits on two 80 GB GPUs. Each of these choices trades memory against accuracy and throughput, so the ML team should test the quantized model before it is used for sizing.

This calculation is the basis of an inference requirement. Multiply GPUs per instance by the number of replicas needed for your peak request rate and availability target, then add headroom for rolling updates.

Partitioning Smaller Workloads

Notebooks, embedding models, and small inference services rarely need a full GPU. NVIDIA Multi-Instance GPU (MIG) splits an H100 into as many as seven isolated instances of about 10 GB each. Requirements should state which workloads can share GPUs this way, because partitioning changes the GPU count significantly.

Layer 2: Storage Requirements

AI storage has more distinct jobs than most enterprise storage, and each should have its own requirement.

A full checkpoint of a 70B model with optimizer state is roughly 1 TB. Writing it at 5 GB/s takes about 200 seconds; at 20 GB/s, about 50. That difference is a requirement, not a detail. OneSource Cloud's AI Storage Architecture designs tiered storage for these separate jobs.

Layer 3: Networking Requirements

AI clusters need two networks with different requirements.

The compute fabric connects GPU servers for distributed training. Current reference designs give each GPU its own 400 Gb/s port, as in DGX H100-class systems with eight ConnectX-7 adapters per server, on a non-blocking InfiniBand or RoCE v2 fabric. Inside each server, GPUs communicate over NVLink at 900 GB/s per GPU.

The front-end network carries user access, inference traffic, storage traffic, and connections to corporate systems. Its requirements are about segmentation, latency to applications, and bandwidth to storage.

Inference-only environments often do not need a training-grade fabric. Writing separate requirements for each network avoids paying for a fabric that inference workloads will not use. See AI Networking Services.

Layer 4: Orchestration Requirements

Orchestration requirements answer who can use which GPUs, for what, and how that is enforced. Specify at least:

• Quota model: reserved and borrowable GPU capacity per team or project
• Workload types supported: distributed training, batch jobs, notebooks, and autoscaling inference on one platform
• Scheduling features: gang scheduling for multi-node jobs, preemption rules, and backfill for small jobs
• Self-service: time for a new data scientist to get a working notebook without a ticket
• Usage reporting: GPU-hours by team and project, exportable for chargeback

The OnePlus Platform, OneSource Cloud's AI orchestration platform, provides GPU scheduling, quotas, Kubernetes lifecycle management, and a developer hub with Jupyter and Kubeflow on dedicated clusters. See the OnePlus Platform.

Layer 5: Security and Compliance Requirements

Security requirements should be written per AI artifact, not just per system, because AI pipelines create new copies of sensitive data.

Regulated organizations then map these controls to their frameworks: HIPAA for protected health information, SOC 2 for service organizations, and GDPR where EU personal data is involved. Infrastructure supports compliance; it does not deliver it alone. The organization still owns its policies, risk assessments, and agreements such as business associate agreements.

Layer 6: Operations Requirements

Operations is the layer most often missing from requirement documents, and it is where GPU clusters lose performance over time.

Requirements should name who performs each task. If the answer is "the platform team" and that team has two people, the requirement is not met. Managed AI Infrastructure from OneSource Cloud covers 24/7 operations, monthly SLA reporting, incident response, MIG configuration, and cost tuning.

Who Owns Each Layer Under Different Deployment Models

OneSource Cloud's Private AI Infrastructure follows the third model, with dedicated environments in U.S. data centers, including its facility in Richardson, Texas.

Checklist: What to Put in an Enterprise AI Infrastructure RFP

1. Workload inventory with model sizes, training methods, token volumes, request rates, and latency targets
2. GPU memory per model instance, calculated as shown above, and replicas per service
3. Training GPU-hours per month and the largest single job size
4. Storage throughput for data reads and checkpoint writes, plus capacity by tier
5. Compute fabric specification (bandwidth per GPU, topology) or a statement that inference does not need one
6. Quota, scheduling, and self-service requirements
7. Data classification and residency requirements per artifact type
8. Compliance frameworks and the split of responsibilities with the provider
9. Operations tasks, response times, and reporting
10. Acceptance tests: GPU diagnostics, NCCL bandwidth tests, storage benchmarks, and a representative workload run
11. Exit terms: data and model export, decommissioning, and certificate of data destruction

FAQ

What are the main components of enterprise AI infrastructure?

Six layers: GPU compute, storage, networking, orchestration, security and compliance controls, and operations. Production problems usually come from the layers around the GPUs, such as slow checkpoint storage or missing quotas, rather than from the GPUs themselves.

How much GPU memory do I need to serve a 70B LLM?

In BF16, the weights need about 140 GB. The KV cache adds about 0.33 MB per token, so 32 concurrent requests at 8,192 tokens add about 86 GB. That totals roughly 226 GB, or four 80 GB GPUs. FP8 quantization can bring it down to about two GPUs.

Do enterprises need InfiniBand for AI?

Only for distributed training across servers. Inference and single-server fine-tuning can run well on standard high-speed Ethernet. Write fabric requirements based on the largest training job, not by default.

What is the difference between enterprise AI infrastructure and a cloud GPU instance?

A cloud GPU instance provides compute. Enterprise AI infrastructure adds governed storage, a designed network, multitenant orchestration, artifact-level security controls, and defined operations, with a named owner for each.

How do enterprises charge AI infrastructure costs back to teams?

Most use GPU-hours by team or project, collected from the scheduler, multiplied by an internal rate that covers hardware, operations, storage, and networking. Some add a premium for reserved capacity compared with borrowed capacity.

Summary

Enterprise AI infrastructure requirements should be written per layer and per stage. Compute requirements come from calculated GPU memory per model instance and GPU-hours per month. Storage requirements separate data reads, checkpoints, registries, RAG stores, and logs. Networking separates the training fabric from the front-end network. Orchestration, security, and operations each need explicit requirements and a named owner. A requirement document built this way makes vendor proposals comparable and production readiness measurable.

To turn your workload list into an infrastructure specification, explore OneSource Cloud's Private AI Infrastructure.