How to Evaluate LLM Inference Providers: Metrics That Matter

"Which LLM inference provider is best?" produces leaderboards, and leaderboards produce regret — because provider performance is condition-dependent, pricing is structurally different across vendors, and the number that decides your experience is yours, not the benchmark's. The responsible version of the question is a framework: four metrics read at your operating conditions, testing across conditions rather than once, pricing normalized before comparison, and evidence requests that separate measurement from marketing. This page provides it.

The Four Metrics That Decide Performance

Four metrics decide inference provider performance — time to first token (the perceived-speed number), throughput in output tokens per second, uptime as measured history rather than SLA promise, and quantization fidelity (what the served model actually is) — and each only means something at stated operating conditions.

The four-metric convergence comes from platform guidance that evaluates providers for a living — the routing platforms see every provider's real behavior, and their evaluation settles on exactly these numbers — with platform documentation providing the metric reference (TTFT, inter-token latency, throughput, requests per second). The fidelity row is the sleeper: two providers serving "the same model" at different quantizations are serving different quality, and the disclosure of what runs beneath the API is an evaluation requirement, not a courtesy.

Test Across Conditions, Not Once

Evaluate across conditions because performance varies with load, prompt length, and concurrency: run each candidate at your prompt-length mix across a concurrency ladder, read throughput at your latency SLO, and treat differences inside run variance as ties — the same manifest discipline the site's benchmark-methodology page defines, applied to vendors.

1. Build the condition matrix: your prompt-length distribution crossed with a concurrency ladder from below to above target load — conditions from your traffic, never the vendor demo's.
2. Run each candidate on it: identical prompts, identical schedule, holdout data the vendor cannot have seen.
3. Read at your SLO: capacity is throughput where your latency percentile holds — peak-throughput numbers past your SLO are trivia.
4. Apply the tie rule: differences inside observed run-to-run variance are ties, and vendor guidance is blunt about why single benchmarks mislead — load, prompt length, and concurrency move every number.

The multi-condition discipline exists because the failure it prevents is expensive: the provider that won the single benchmark and lost your traffic pattern, discovered in production. The condition matrix is also the negotiation artifact — results at stated conditions are comparable across vendors and repeatable on request, which is exactly what the evidence step asks for.

Pricing Structures and the Evidence Request

Normalize pricing before comparing: cost per million tokens at your volume, batching-discount behavior, and throughput-normalized cost (what a dollar buys at your SLO) differ structurally across providers — then request the evidence: uptime history, quantization disclosure, rate-limit terms, and benchmarks at stated conditions, with unproducible claims treated as marketing.

The throughput-normalized row is the one vendors least enjoy and buyers most need: per-token prices mean nothing if the provider's throughput at your SLO is half the competition's, because your cost per delivered experience doubles. Then the evidence request closes the evaluation: uptime measured over windows rather than promised in SLAs, quantization disclosed per model, rate limits that will not surprise your peak, and benchmarks reproducible at stated conditions — claims that cannot produce these artifacts are marketing, and the framework's job is complete when the shortlist is decided on produced evidence rather than published numbers.

FAQ

Which metric matters most for an inference provider?

Whichever one your users feel: interactive products live and die on time-to-first-token and inter-token latency, batch workloads on throughput and cost — which is why the four-metric set gets read at your conditions rather than ranked in the abstract.

Can we trust a provider's published benchmarks?

As priors, not answers: published numbers reflect their conditions, and performance varies with load, prompt mix, and concurrency — run the multi-condition evaluation on your traffic before committing, treating vendor benchmarks as the hypothesis your test confirms or kills.

Is choosing an inference provider the same as choosing infrastructure?

Adjacent but distinct: providers sell serving performance (metrics, model fidelity, per-token economics) while infrastructure sells capacity (GPUs, boundaries, commitments) — API-first estates answer the provider question, self-hosting estates answer the infrastructure question, and hybrids answer both.