How to Benchmark LLM Inference Before Committing to a GPU Cloud

Choosing an inference platform on quoted numbers is how teams end up surprised in production: vendor benchmarks run on someone else's workload, at someone else's concurrency, with someone else's pricing basis. The reliable alternative is a benchmark you run yourself, designed so its results are comparable across candidates and defensible in procurement. This page provides that methodology: a run manifest that pins the workload, an environment-parity procedure, a concurrency sweep read at your latency SLO, cost normalization on the same basis, and the repetition and trust checks that separate signal from noise.

Prerequisites: Define Metrics and Pin the Workload

Fix the metric family (TTFT, per-token latency, throughput at SLO, request rates) and freeze a workload profile: model and precision, prompt and output length distributions, context lengths, sampling parameters, and dataset, all recorded in a run manifest.

A benchmark measures only what its manifest specifies — everything left unspecified is silently not being measured. Before touching any environment, fix:

• The metric family. Time to first token (TTFT), per-token generation latency (inter-token latency or time per output token), request throughput and token throughput, and latency at stated percentiles. Vendor guidance on LLM benchmarking treats this family — throughput and latency measured together, tied to cost — as the core of any evaluation.
• The workload profile: model and precision or quantization; prompt and output length distributions; context lengths; sampling parameters (fixed, typically temperature 0 for comparability); and the dataset or prompt source, recorded in full.
• A holdout dataset not shared with any vendor being evaluated, so results cannot be tuned against your visible data.

Write all of it down as a run manifest — engine and model versions, every distribution parameter, the sampling settings, dataset construction, and the date. The manifest is what makes a number auditable three months later and comparable across two providers; without it you have anecdotes.

Step 1: Reproduce the Same Configuration Everywhere

Deploy the same model, engine version, quantization, and configuration on each candidate, verify with loaded-model checks (version, precision, memory), and run warmup until steady state before recording.

Comparability comes from controlled variables. For each candidate environment:

1. Deploy the same model at the same precision on the same engine version where you control the stack. Pin versions explicitly — engine behavior changes across releases.
2. Verify the loaded state: confirm the served model, quantization, and memory configuration through the engine's reporting rather than assuming the deployment script worked.
3. Warm up before recording. Run load until steady state (compilation, caching, and allocation settle) and discard the warmup window.
4. Log the environment: hardware type, driver and engine versions, configuration flags, and anything the environment fixed that you could not change.

Vendor-managed platforms sometimes hide tunables — batching policies, memory settings, scheduler parameters. Either get them exposed and pinned, or record the limitation and accept that you are comparing that vendor's offered configuration rather than an ideal one. What you cannot do is compare your tuned self-hosted stack against their default settings and call the result fair in either direction.

Step 2: Sweep Concurrency and Read the Curves

Run a concurrency ladder from well below to above target load, record the throughput-latency curve at each step, and read capacity at your SLO percentile rather than at peak throughput, which typically occurs past acceptable latency.

Serving performance is load-dependent, so a single concurrency point cannot characterize an environment. Run a concurrency ladder — from well below target load to above it — and record the full throughput-latency curve at each step. Then apply the reading rule that most evaluations miss:

Capacity is the throughput available at your latency SLO, not the peak of the curve. Peak throughput typically occurs past acceptable latency, when TTFT and per-token latency have already degraded beyond what users or downstream systems tolerate. The useful operating point is where your chosen percentile (for example, p95 TTFT or p99 per-token latency) still holds.

When a vendor quotes a single number, they are quoting one point on a curve you have not seen, measured on a workload that is not yours. The sweep is how you see the curve.

Step 3: Normalize by Cost at Equal Quality

Compute cost per million tokens at the concurrency that meets the SLO, on the same date, with the same pricing units, and record the configuration that achieved it; performance without unit economics is not a decision.

Performance numbers become procurement inputs only when converted to a common cost basis:

1. Compute cost per million tokens at the SLO point — the throughput-at-SLO from Step 2, divided by the environment's price at the same date and pricing basis (on-demand, reserved, or committed).
2. Hold quality constant. The comparison is only valid at equivalent model precision and output quality; a cheaper quantization that fails your quality bar is not a cheaper option.
3. Stamp the date and basis. GPU pricing moves; a comparison without its date and pricing basis has a shelf life measured in weeks.

The table's discipline matters more than its cells: same SLO, same quality, same date, same pricing basis. Mixed-basis comparisons are the most common way procurement decks mislead — usually unintentionally.

Verify: Repetition, Variance, and Trust Checks

Repeat runs and report medians with spread, monitor GPU utilization and batch composition during tests, defeat caching distortions with fresh or hashed prompts, and require raw logs so results can be reconstructed by a third party.

Two properties of LLM serving make single runs untrustworthy: documented non-determinism (identical requests vary because batch-dependent GPU kernels produce slightly different results under dynamic batching), and ordinary run-to-run variance in shared environments. The verification layer:

• Repeat each concurrency point several times and report medians with spread (min/max or interquartile range). Differences smaller than observed spread are ties, not wins.
• Monitor during the test: GPU utilization, batch composition, and queue depth from the environment's own metrics — confirming the load you believe you applied is the load that occurred.
• Defeat caching distortions. Prompt caches can make repeated or similar requests dramatically faster; use fresh or hashed prompt variants so results reflect serving, not cache hits.
• Require raw logs. For vendor-run evaluations, request the underlying event logs so a third party could reconstruct the summary — the same reconstruction standard an internal audit would apply.

Declare in advance the minimum delta you will treat as signal — for example, "we act only on differences exceeding our measured run-to-run spread." Written after seeing results, that threshold bends to whatever the numbers happened to do.

Serving Decision Matrix: Enterprise LLM Inference Infrastructure

When the benchmark identifies capacity requirements that justify dedicated infrastructure, candidates that passed can be evaluated for deployment — including on private AI infrastructure run under your own boundary — using the same manifest discipline so the production environment matches the tested one.

FAQ

Can we rely on public benchmarks like MLPerf instead of running our own?

Use them to shortlist, not to decide. Public suites compare reference workloads under fixed submission rules, which makes them comparable — but your prompt lengths, context mix, concurrency pattern, and SLO differ from the reference configuration. Final decisions need an own-workload run under the manifest discipline in this article; the suite tells you who is credible, your benchmark tells you who is right for you.

What vendor tactics inflate benchmark results?

The recurring four: pre-warmed prompt caches that turn the benchmark into a cache test; datasets chosen to match the engine's strengths; peak-throughput numbers quoted past your latency SLO; and unspecified quantization quietly lowering quality while raising throughput. The manifest, cache-busting inputs, SLO-anchored reading, and quality-pinned cost normalization neutralize each in turn.

How many runs make a benchmark trustworthy?

Enough to separate signal from variance: repeat every concurrency point several times, report medians with spread, and treat differences inside the observed run-to-run spread as ties. The count matters less than the discipline of deriving your decision threshold from measured variance rather than from the result you preferred.

Why deploy latency-sensitive LLM inference on OneSource private GPUs?

OneSource private GPU infrastructure delivers 100% dedicated bare-metal compute and VRAM, completely isolated from cross-tenant contention. This eliminates hypervisor scheduling jitter and shared-network packet collisions, ensuring deterministic P99 tail latency, sustained token throughput, and optimal tensor parallel scaling for production enterprise LLM serving.