How to Calculate Cost per Token for Production LLM Inference

Cost per token for LLM inference is the total cost of serving a model divided by the number of tokens it processes, expressed as a per-token figure that lets teams price, budget, and compare inference options. It answers a simple question: what does one token of real traffic cost across the whole infrastructure?

This is the practical conversion of GPU capacity, utilization, and request volume into a number usable for budget and unit-economics decisions.

Gather the Cost Inputs

Before any division, collect the full monthly cost of running inference. Include the GPU capacity the workload actually needs, storage for weights, vectors, and logs, networking, and the operations cost of keeping the service up, whether internal staffing or a managed fee. Use monthly figures so the calculation is easy to update.

The accuracy of the result depends on these inputs. Understate capacity or operations and the per-token number will be optimistic and produce a budget that is already wrong.

Estimate the Token Volume

Estimate how many tokens the service processes per month from the traffic that will actually run against it. Multiply projected requests by the average tokens per request, separating input tokens and output tokens because generation cost usually dominates. Include realistic volume, not the launch-day optimistic case, and account for retries and regeneration that add tokens.

Volume is the denominator, so a realistic estimate matters as much as an accurate cost numerator.

Convert to a Per-Token Figure

With total cost and total token volume, the cost per token is simply the monthly cost divided by the monthly token count. Producing the number in isolation is less useful than showing its parts, so split it into the fraction that comes from GPU capacity and the fraction from token-dependent volume. That split reveals whether the lever is raising utilization or reducing waste.

Use the number as a planning input, not a fixed price. A cost per token at low utilization is higher than the same workload at higher utilization, because the same fixed GPU cost is spread over fewer tokens. Recompute it as utilization and volume change.

Compare Self-Hosted vs Managed Cost Per Token

The same method compares approaches. A self-hosted model must absorb capacity that may sit idle between bursts, raising the per-token cost unless utilization is high. A managed service bundles utilization, operations, and capacity into its price. Convert each to a per-token number over the same volume and utilization assumptions to compare fairly.

1. Collect total monthly cost: capacity, storage, networking, and operations.
2. Estimate token volume: requests multiplied by input and output tokens, including retries.
3. Divide for the per-token figure: monthly cost divided by monthly tokens.
4. Split the drivers: separate the fixed-capacity fraction from the token-dependent fraction.
5. Recompute with utilization: update the number as utilization and volume change.

Private AI infrastructure can place capacity for a predictable per-token number, while managed AI infrastructure bundles utilization and operations so the per-token figure is easier to hold. An architecture review can produce the utilization and capacity assumptions for a realistic number.

FAQ

Why does cost per token vary so much?

Cost per token is highly sensitive to utilization. The same fixed GPU cost spread over many tokens yields a low per-token number; the same cost over few tokens (or a cluster with idle capacity) yields a high one. It also varies with model size, token mix, and operations cost. Compare approaches at the same utilization and volume assumptions, not across different utilization levels.

What is the formula for cost per token?

Cost per token equals the total monthly inference cost divided by the total monthly token volume. Total cost includes GPU capacity, storage, networking, and operations; total volume is requests multiplied by input and output tokens, including retries. The formula is a budgeting tool, and its accuracy depends on realistic cost and volume inputs.

Is it cheaper to self-host LLM inference?

Not necessarily. Self-hosting avoids per-token API premiums but adds capacity, storage, and operations cost, and idle capacity during low usage raises the real per-token figure. Managed or API options bundle utilization and operations into the price. Compute the per-token cost for your volume and utilization in both cases before deciding.

How can I reduce the cost per token?

Raise GPU utilization so fixed capacity is spread over more tokens, right-size the cluster to measured demand, reduce waste from idle nodes and unnecessary regeneration, and choose the operations model that fits. The per-token figure falls when the same fixed cost serves more completed tokens rather than when hardware cost alone is minimized.

Summary

Cost per token for LLM inference is total monthly cost divided by total token volume, with the drivers separated into capacity and token-dependent components. It is a planning number that only makes sense at a known utilization level. Recomputing it as utilization and volume change keeps the budget honest and reveals whether the lever is utilization or waste.