How Many GPUs to Serve a 70B Model in Production

The 70-billion-parameter model size (epitomized by Meta's Llama-3.1-70B and Qwen-2.5-72B) has emerged as the undisputed enterprise sweet spot: delivering reasoning, coding, and synthesis capabilities rivaling proprietary commercial APIs while remaining compact enough for private deployment. However, moving a 70B model from research prototyping to user-facing enterprise production introduces complex hardware sizing decisions. Engineering leaders frequently ask: How many GPUs are actually required to serve a 70B model? Can it run on two 80GB GPUs? Does production require four or eight accelerators? Sizing hardware for a 70B model depends not merely on holding model weights in VRAM, but on accommodating the Key-Value (KV) cache for concurrent users, maintaining low Time-to-First-Token (TTFT), and hitting generation latency budgets. On OneSource Cloud bare-metal infrastructure, high-bandwidth SXM5 configurations provide the ideal environment for high-throughput 70B model serving.

70B Hardware Configuration Comparison Matrix

Mathematical Sizing Formula for 70B VRAM Allocation

Total required VRAM ($M_{\text{total}}$) is the sum of static model parameters and dynamic multi-tenant KV cache memory:

$$M_{\text{total}} = \frac{P_{\text{model}} \times B_{\text{precision}}}{1024^3} + N_{\text{users}} \times \left( 2 \times L_{\text{context}} \times N_{\text{layers}} \times N_{\text{kv\_heads}} \times D_{\text{head}} \times P_{\text{kv\_bytes}} \right) \times 1.15$$

For Llama-3-70B in FP16 ($P = 70\times 10^9, B = 2\text{ bytes}$), static weights consume $140\text{ GB}$. With $N_{\text{layers}} = 80$, $N_{\text{kv\_heads}} = 8$, $D_{\text{head}} = 128$, each 8,192-token context in FP16 consumes:

$$M_{\text{KV per user}} = 2 \times 8,192 \times 80 \times 8 \times 128 \times 2 = 2.68\text{ GB per user}$$

To support 50 concurrent users at full 8k context requires $140\text{ GB (Weights)} + 134\text{ GB (KV Cache)} = 274\text{ GB VRAM}$. This strictly exceeds the capacity of two 80GB GPUs ($160\text{ GB}$ total), mathematically proving that four 80GB GPUs ($320\text{ GB}$ total) are the minimum requirement for production concurrency. OneSource Cloud provides 8x H100 SXM5 instances with 640GB aggregate VRAM, providing abundant memory headroom.

Production Python Script: 70B GPU Sizing Calculator

The following script calculates exact VRAM requirements, recommended GPU counts, and predicted token generation speeds for 70B deployments:

import math
def size_70b_cluster(
    concurrent_users: int,
    avg_context_tokens: int,
    weight_precision: str = "FP16",
    gpu_vram_gb: float = 80.0
) -> dict:
    weight_bytes = 1.0 if weight_precision == "FP8" else 2.0
    weights_gb = (70.0 * 1e9 * weight_bytes) / ((1024 * 1024 * 1024))
    // Llama-3-70B architecture parameters
    num_layers = 80
    num_kv_heads = 8
    head_dim = 128
    kv_bytes = 1.0 if weight_precision == "FP8" else 2.0
    // Per-user KV cache footprint
    kv_cache_per_user_gb = (2 * avg_context_tokens * num_layers * num_kv_heads * head_dim * kv_bytes) / ((1024 * 1024 * 1024))
    total_kv_gb = concurrent_users * kv_cache_per_user_gb
    total_vram_required = (weights_gb + total_kv_gb) * 1.15  // 15% CUDA overhead
    // Minimum GPUs needed to fit VRAM
    gpus_needed = math.ceil(total_vram_required / gpu_vram_gb)
    // Round to valid Tensor Parallel configurations (1, 2, 4, 8)
    valid_tp = [1, 2, 4, 8]
    recommended_gpus = next((tp for tp in valid_tp if tp >= gpus_needed), 8)
    return {
        "weight_footprint_gb": round(weights_gb, 2),
        "total_active_kv_gb": round(total_kv_gb, 2),
        "total_vram_required_gb": round(total_vram_required, 2),
        "recommended_gpu_count": recommended_gpus,
        "recommended_topology": f"{recommended_gpus}x {int(gpu_vram_gb)}GB GPUs (TP={recommended_gpus})"
    }
// Run sizing for 60 concurrent users with 8k average context
res = size_70b_cluster(concurrent_users=60, avg_context_tokens=8192, weight_precision="FP16")
print("70B Model GPU Sizing Audit:")
for k, v in res.items():
    print(f"  {k}: {v}")

Why Inter-GPU Bandwidth (NVLink) Is Decisive

When running a 70B model partitioned across four or eight GPUs using Tensor Parallelism (TP=4 or TP=8), every single transformer layer requires distributed All-Reduce operations across GPUs. On standard PCIe slots without NVLink, communication bandwidth is throttled to 64 GB/s, causing token generation speed to collapse below 10 tokens/sec. On NVIDIA SXM5 nodes with high-speed NVLink delivering 900 GB/s bidirectional bandwidth, inter-GPU communication is near-instantaneous, enabling Llama-3-70B to generate over 60 tokens/sec per user stream. On OneSource Cloud dedicated bare-metal servers, native SXM5 NVLink topologies ensure that multi-GPU tensor parallelism achieves full theoretical throughput.

Frequently Asked Questions

Can I serve a 70B model on consumer GPUs like the RTX 4090?

Running a 70B model across multiple 24GB RTX 4090 GPUs is possible for hobbyists using 4-bit quantization (EXL2/AWQ). However, lack of high-speed NVLink and limited PCIe bandwidth results in sluggish generation speeds and zero enterprise concurrency headroom, making it unsuitable for production APIs.

What is the throughput difference between TP=4 and TP=8 on H100?

TP=4 provides lower inter-GPU communication overhead for small batch sizes, offering lower latency per individual user. TP=8 provides double the aggregate VRAM and HBM memory bandwidth, enabling substantially higher batch concurrency and higher total cluster throughput.

How does OneSource Cloud optimize 70B model serving?

OneSource Cloud delivers dedicated 4-GPU and 8-GPU H100 SXM5 instances with 900 GB/s NVLink fabrics and enterprise NVMe storage, allowing enterprise teams to deploy vLLM or TensorRT-LLM with maximum concurrency and low latency.