H100 Capacity for 70B LLM Inference by Precision

H100 capacity planning for a 70B LLM is a memory and throughput calculation that combines model weights, KV cache, runtime workspace, concurrency, and service objectives. Weight precision sets the minimum memory floor, but it does not determine a production GPU count by itself.

An 80 GB H100 can theoretically hold roughly 70 GB of one-byte-per-parameter weights or 35 GB of four-bit weights before overhead. Production deployments need additional memory for quantization metadata, KV cache, communication buffers, kernels, fragmentation, and operational headroom, so the usable configuration must be benchmarked against real traffic and service objectives. Test failover capacity as well. Document assumptions.

Start With the Weight Memory Floor

A 70-billion-parameter dense model requires approximately parameter count multiplied by bytes per stored parameter. This simple estimate excludes every runtime allocation, but it shows why precision changes the feasible topology.

NVIDIA lists 80 GB of memory for the H100 SXM and 94 GB for H100 NVL. Do not use those capacities interchangeably in a plan. Record the exact board, server topology, interconnect, power configuration, runtime, and quantization implementation.

Add KV Cache for Context and Concurrency

The KV cache stores attention state for active sequences. Its size grows with the model architecture, cached token count, concurrent sequences, and cache precision. Longer prompts, longer generated outputs, and more simultaneous users can make KV cache the limiting resource even when quantized weights fit easily.

Use the deployed model configuration to calculate bytes per cached token, then multiply by the maximum tokens retained across active sequences. Include prefix caching, speculative decoding, beam search, and runtime block allocation when used. Do not estimate cache from the model name alone because architectures can use different attention and KV-head designs.

Reserve Runtime and Reliability Headroom

Inference engines allocate memory for kernels, temporary tensors, communication, CUDA graphs, quantization metadata, and memory management. Fragmentation and version changes can alter the usable limit. A configuration that loads with only a few free gigabytes may fail under longer prompts, peak concurrency, or a runtime update.

Set an explicit headroom policy and test the maximum supported request shape. Monitor out-of-memory events, cache occupancy, evictions, preemptions, and allocation failures. Admission control should reject or queue work before memory exhaustion destabilizes the service.

Translate Memory Fit Into a Practical GPU Range

The following ranges are planning starting points, not performance guarantees:

• FP16 or BF16: two 80 GB GPUs meet only the raw weight floor. Four GPUs provide a more workable starting point for modest KV cache and runtime headroom, while higher concurrency or throughput can require additional replicas or wider parallelism.
• INT8: one 80 GB GPU is too tight for a dependable weight-plus-runtime plan in many implementations. Two GPUs are a more realistic baseline when quality has been validated and the runtime supports the quantization path.
• 4-bit: one 80 GB H100 can be feasible for low-concurrency inference, but a second GPU or additional replica may be justified by context length, throughput, availability, and quality requirements rather than weight memory.

H100 NVL has a different memory capacity and topology, so rerun the calculation for that platform. The number of GPUs in one model replica should also be separated from the number of replicas required for traffic and availability.

Choose Parallelism Around the Service Objective

Tensor parallelism splits model work across GPUs and can make large weight sets fit, but it introduces communication. Pipeline parallelism changes execution flow and is not automatically best for interactive latency. Multiple smaller replicas can improve throughput and fault isolation when each replica fits, while one wider replica may support larger memory requirements.

Benchmark time to first token, inter-token latency, tokens per second, queue time, and errors at expected prompt and output lengths. Include peak concurrency and mixed request sizes. The preferred topology is the smallest reliable configuration that meets latency and throughput objectives with recovery headroom, not the smallest configuration that starts.

Account for Quality and Operational Constraints

Quantization can lower memory use, but the acceptable method depends on model, task, runtime, and quality threshold. Validate accuracy or task success on representative prompts before turning a memory saving into a capacity decision. Also confirm that the quantized format is supported by the selected inference engine and H100 kernels.

Capacity planning should include rolling upgrades, maintenance, and failure. If a service requires availability during one-GPU or one-node loss, the steady-state cluster must have enough spare capacity to absorb traffic without violating latency objectives.

Run an Acceptance Benchmark Before Procurement

1. Fix the model artifact. Record exact weights, precision, quantization method, tokenizer, and runtime version.
2. Define traffic. Use production-like prompt lengths, output lengths, concurrency, streaming, and arrival bursts.
3. Test several topologies. Compare tensor parallel size, replica count, cache allocation, and batching policy.
4. Measure quality and service. Check task quality, latency percentiles, throughput, memory, errors, and recovery.
5. Apply growth and failure headroom. Size the committed environment for forecast demand and the stated availability objective.

OneSource Cloud's Private AI Infrastructure can be designed around dedicated GPU topology, storage, and networking. Managed AI Infrastructure adds performance validation and ongoing operations for production serving environments.

FAQ

Can a 70B model run on one H100 80 GB GPU?

It can be feasible with an appropriate low-bit quantization, limited context and concurrency, and a compatible runtime. FP16 or BF16 weights do not fit on one 80 GB GPU. Even when quantized weights fit, benchmark KV cache, runtime overhead, quality, and latency before treating one GPU as production capacity.

Are two H100 GPUs enough for FP16 70B inference?

Two 80 GB H100 GPUs provide 160 GB of aggregate memory, which is only modestly above the approximate 140 GB raw FP16 weight floor. Runtime state and KV cache can make that configuration too constrained. Use it as a weight-fit boundary, not an automatic production recommendation.

Why does context length change the H100 count?

Longer active sequences require more KV-cache memory. At production concurrency, cached tokens across many requests can consume substantial capacity and reduce the room available for batch growth. Context also increases prefill work, so both memory and latency must be tested at the supported limit.

Should GPU count be based on memory or throughput?

Use memory to eliminate infeasible configurations, then use throughput, latency, availability, and growth to choose capacity. A model may fit in memory but fail the service objective. Conversely, additional replicas may be needed for traffic and recovery even when one replica has ample memory.

Summary

Estimate H100 capacity for 70B inference by calculating weight memory, KV cache, runtime overhead, concurrency, and headroom, then benchmark production-like traffic and quality. Teams can start with a OneSource Cloud AI Cluster Survey to translate model and service requirements into a validated GPU topology.