LLM Inference GPUs Compared: A100, H100, H200, or B200 for Production

Four NVIDIA generations are in production inference service right now — A100, H100, H200, and B200 — and each has a distinct character that makes it the right answer for a different workload. Pairwise comparisons abound; what procurement actually needs is the four-way frame: what each generation is known for, which constraint (memory, compute, budget) binds your workload, and the verification steps before committing. This page provides that selector from public specifications and benchmark coverage current as of October 2026.

Four Generations, Four Characters

Public coverage converges on four characters: A100 — the previous generation still sensible for budget batch inference and legacy stacks; H100 — the production workhorse frequently cited as the best cost-per-token part; H200 — the H100's compute with much larger memory, winning when models are memory-bound; B200 — the current performance ceiling with FP4 support, justified when token throughput pays its premium.

The memory column tells the structural story of the last three years: capacity per card nearly tripled from A100 to B200, which is why the generations differ less on "speed" than on what fits per GPU — and therefore how many GPUs a given model needs. Every figure above is from public specifications and benchmark coverage, labeled as published findings rather than our measurements; procurement re-verifies both specification and price at decision time, because both move quarterly.

Matching Workloads to Generations

Match on the binding constraint: small-to-mid models and batch tolerance point at A100 economics, general production serving points at H100, large models whose KV cache or weight footprint exhausts 80GB point at H200's capacity (vendor benchmarks show it serving large models roughly twice as fast as H100 when memory-bound), and frontier throughput requirements point at B200 where the premium is recovered in tokens.

The matching rule is constraint-first, not generation-first: identify what actually binds your serving — memory (model size plus KV cache at concurrency), compute (tokens per second demanded), or budget — and the table picks the generation that removes that constraint. Two fleet realities complicate the clean version: quantization choices move workloads between the memory-bound and compute-bound rows, and mixed fleets legitimately span generations, with H100s carrying the general band while H200s or B200s serve the constrained workloads that justify them.

Before You Commit: The Verification Steps

Verify three things at decision time: availability and current pricing for your region (both move quarterly), your model's benchmark on the shortlisted part under your own manifest, and the pairwise deep dive for the two finalists — the selector narrows, the pairwise comparisons and your benchmark decide.

1. Check availability and pricing for your part and region: coverage reports newest-generation capacity scarcest, with wait times varying by part — availability is part of the decision, not a footnote.
2. Benchmark your model under your manifest: your quantization, your context lengths, your concurrency ladder — published numbers are priors, and this page is deliberately a prior.
3. Read the pairwise deep dive for the finalists: when the selector narrows to two parts, the detailed head-to-heads (H100 versus A100, H100 versus H200, B200 versus H200 — each covered in depth on this site) carry the per-dimension detail this selector's width cannot.

The chain — selector to pairwise to your benchmark — is the discipline that keeps procurement honest: each stage narrows on evidence, and the final call is always your workload's measured number rather than any published table, including this one.

FAQ

Is the A100 still worth buying for inference in 2026?

In its narrow lane, yes: budget batch inference on small-to-mid models and estates pinned to legacy stacks — but every new commitment should carry an end-of-life plan, because the lane narrows as serving stacks optimize for FP8 and later formats the A100 lacks.

Should we skip a generation and go straight to the newest part?

Only when your workload monetizes what the newest generation uniquely provides — FP4 throughput or frontier-model memory — because skipping means paying the scarcity premium without the utilization that pays for it; the binding-constraint match, not generation counting, should drive the order.

H100 or H200 for serving a 70B model?

Memory-bound serving of 70B-class models is exactly where H200 wins — vendor benchmarks show roughly 1.9x H100 throughput there — so if the model's weights plus KV cache pressure 80GB, H200 is the match; if you serve quantized variants comfortably within 80GB, H100 holds the value edge.