H100 vs A100 for LLM Inference: Which GPU Fits Your Workload

For LLM inference, the H100's advantages over the A100 — higher memory bandwidth, a dedicated transformer engine, FP8 support, and faster interconnects — translate into higher throughput per GPU for large models, while the A100 remains competitive for smaller models and cost-constrained workloads where its lower hourly price wins on cost per token. The right choice is not which GPU is faster on paper, but which delivers the best cost per token for your specific model size, traffic pattern, and latency target.

The comparison matters because inference, unlike training, is a steady-state cost that compounds with usage. A GPU choice that is 20% faster but 50% more expensive can be either a win or a loss depending on utilization. Teams that pick by peak specs alone often overpay; teams that pick by hourly price alone often underperform on latency. The decision requires mapping GPU characteristics to the workload's actual demands.

This guide compares the H100 and A100 on the dimensions that decide inference economics: memory and bandwidth, the transformer engine, interconnect, throughput, and cost per token. It ends with a workload-based recommendation for when each GPU wins.

Why Inference Stresses Different GPU Features Than Training

Training is compute-bound: it spends most of its time doing large matrix multiplies across many GPUs. Inference is different. For autoregressive LLM serving, each generated token depends on the previous one, so the workload is largely memory-bandwidth-bound — the GPU spends its time moving weights and KV cache through memory, not doing peak math. This is why memory bandwidth, not raw FLOPS, is often the decisive spec for inference throughput.

This shifts the H100-versus-A100 comparison. The H100's higher peak FLOPS helps, but its larger memory bandwidth advantage is what most directly improves tokens per second for autoregressive serving. For workloads that batch many requests, the bandwidth advantage compounds, because a batched forward pass reuses weights across requests, making memory movement even more dominant.

Memory and Bandwidth

Memory capacity determines which model fits on a single GPU or how many GPUs a model needs, and memory bandwidth determines how fast weights and KV cache move during generation. The H100 SXM offers substantially more memory bandwidth than the A100, which directly raises tokens per second for bandwidth-bound autoregressive inference. Capacity is more nuanced: both GPUs have 80GB HBM variants, so for models that fit in 80GB, capacity is comparable, but the H100's higher bandwidth extracts more throughput from the same memory.

For models larger than 80GB, neither GPU holds the model alone, so the decision shifts to multi-GPU tensor parallelism and the interconnect, covered next. For models that fit, the H100's bandwidth advantage is the primary reason it serves more tokens per second per GPU.

The Transformer Engine and FP8

The H100 includes a dedicated transformer engine and hardware support for FP8, a lower-precision format that roughly doubles throughput for eligible operations compared to FP16. For inference, this matters when the workload can use FP8 without unacceptable quality loss. Well-calibrated FP8 inference often preserves quality for typical user queries while substantially increasing tokens per second, which lowers cost per token.

The A100 lacks the transformer engine and FP8, so it serves inference in FP16 or INT8 via quantization. INT8 quantization on the A100 can recover some throughput, but it requires careful calibration and can degrade quality on edge cases. The H100's native FP8 path is generally cleaner and faster for the same quality target. The practical implication: for teams willing to run FP8, the H100's throughput advantage widens considerably.

Interconnect for Multi-GPU Serving

When a model is too large for one GPU, inference uses tensor or pipeline parallelism across multiple GPUs, and the interconnect determines how fast activations move between them. The H100's faster NVLink and support for the latest high-speed fabric reduce the per-step communication cost, which matters for large-model serving where every forward pass crosses the fabric.

For single-GPU serving of models that fit in 80GB, the interconnect is less relevant. For large-model multi-GPU serving, the H100's interconnect advantage compounds with its bandwidth advantage, which is why the H100 pulls further ahead as model size grows. Teams running 70B-class models across multiple GPUs will see a larger H100 benefit than teams serving 7B models on a single GPU.

Throughput and Latency in Practice

Raw throughput, measured in tokens per second per GPU, generally favors the H100 for bandwidth-bound autoregressive serving and widens for larger models and FP8 workloads. But throughput alone does not decide the choice. Two other factors matter: the latency target and the concurrency level.

For latency-sensitive single-request serving (low batch, fast first token), the H100's higher bandwidth and lower-latency memory access improve time-to-first-token and inter-token latency. For high-throughput batch serving (many concurrent requests, tolerance for some latency), both GPUs benefit from batching, but the H100's bandwidth extracts more tokens per second from each batched forward pass. The A100 can still be the right choice when its lower hourly cost produces a better cost per token at the throughput the workload actually needs.

Cost per Token: The Decisive Metric

Cost per token, not peak specs, decides inference economics. Compute it as GPU hourly cost divided by tokens per hour, where tokens per hour comes from real workload benchmarks at your concurrency and latency target. The GPU with the lower cost per token wins for your workload, regardless of which is faster on paper.

The H100 usually wins cost per token for large models, FP8 workloads, and high-throughput serving where its bandwidth and transformer engine are fully used. The A100 often wins for smaller models, lighter traffic, and workloads where the H100's extra capability sits idle — paying for capability you do not use raises cost per token rather than lowering it. The break-even point is workload-specific and should be measured, not assumed.

H100 vs A100 for LLM Inference at a Glance

When the H100 Wins

The H100 wins when its bandwidth, transformer engine, and interconnect are actually used. The clearest cases are large models (70B and above) served across multiple GPUs, where the interconnect and bandwidth advantages compound; FP8 workloads, where the transformer engine roughly doubles eligible throughput; high-throughput batch serving, where the bandwidth extracts more tokens per batched pass; and latency-sensitive serving, where higher bandwidth improves time-to-first-token and inter-token latency.

For these workloads, the H100's higher hourly cost is usually offset by substantially higher tokens per second, producing a lower cost per token. Teams running production serving for large models at meaningful traffic generally find the H100 the better economic choice.

When the A100 Wins

The A100 wins when the H100's extra capability goes unused or when hourly cost dominates. The clearest cases are smaller models (roughly 13B and below) that fit comfortably in 80GB and serve well on a single GPU, where the H100's interconnect advantage is irrelevant; workloads with light or bursty traffic where paying for peak capability raises cost per token; and teams whose budget cannot absorb the H100 premium even where it would technically win.

The A100 is also a sensible choice for development, staging, and non-critical workloads where slightly lower throughput is acceptable. Many teams run a mix: H100 for production serving of large models and A100 for smaller models, experiments, and dev environments. Providers offering dedicated GPU infrastructure with mixed capacity let teams match GPU to workload rather than committing to one generation.

FAQ

Is the H100 worth it for LLM inference?

It depends on the workload. For large models served across multiple GPUs, FP8 workloads, high-throughput batch serving, and latency-sensitive serving, the H100's bandwidth, transformer engine, and interconnect usually produce a lower cost per token despite the higher hourly price. For smaller models, lighter traffic, or budget-constrained workloads, the A100 often delivers a better cost per token because you are not paying for capability you do not use.

Can the A100 80GB run a 70B model for inference?

A 70B model in FP16 does not fit in a single 80GB GPU, so it requires multi-GPU tensor parallelism on both A100 and H100. The difference is that the H100's faster interconnect and higher bandwidth make multi-GPU serving of large models faster and more cost-effective per token. For models that fit in 80GB, the A100 is competitive on capacity; for those that do not, the H100's interconnect advantage matters more.

How much faster is the H100 than the A100 for inference?

It varies by workload. For bandwidth-bound autoregressive serving, the H100's higher memory bandwidth raises tokens per second meaningfully. With FP8 enabled via the transformer engine, eligible operations can roughly double in throughput. The largest gains appear for large multi-GPU models and high-throughput batch serving. Always measure on your specific model and traffic rather than relying on headline multipliers.

Should I upgrade from A100 to H100 for inference?

Upgrade if your workload is large-model serving, FP8-eligible, high-throughput, or latency-sensitive, where the H100's advantages translate into lower cost per token. Do not upgrade if you serve smaller models with light traffic, where the A100's lower hourly cost already wins on cost per token and the H100's extra capability would sit idle. Measure cost per token on both before committing.

How do I choose between H100 and A100 for cost per token?

Compute cost per token as GPU hourly cost divided by tokens per hour, using real benchmarks at your concurrency and latency target. The GPU with the lower cost per token wins for your workload. The H100 usually wins for large models and FP8; the A100 often wins for smaller models and lighter traffic. The break-even point is workload-specific, so measure rather than assume.

Summary

H100 versus A100 for LLM inference is a cost-per-token decision, not a peak-spec decision. The H100 wins for large multi-GPU models, FP8 workloads, high-throughput batch serving, and latency-sensitive serving, where its bandwidth, transformer engine, and interconnect translate into lower cost per token. The A100 wins for smaller models, lighter traffic, and budget-constrained workloads, where its lower hourly cost already delivers the better cost per token. Measure cost per token at your workload's concurrency and latency target before choosing, and consider a mixed fleet that matches GPU to workload rather than committing to one generation.

For teams that want to match GPU to workload without committing to a single generation, dedicated GPU infrastructure with mixed H100 and A100 capacity lets each workload run on the GPU that wins its cost-per-token comparison.