How to Estimate LLM Serving Cost Before Deployment

Estimating LLM serving cost before deployment means modeling four variables — GPU memory required, achievable throughput, expected concurrency, and utilization — to produce a cost-per-token estimate grounded in your workload, not in a vendor's optimistic benchmark. For the cost reduction framework, see how to reduce LLM inference cost. For GPU-level memory and throughput, see how much GPU memory for LLM inference.

The Four Variables That Determine Serving Cost

GPU memory required sets the floor: how many GPUs are needed just to hold the model, the KV cache at the target context length, and the target concurrency. More memory per request means fewer concurrent requests per GPU, which means more GPUs. Achievable throughput per GPU in tokens per second, which depends on memory bandwidth, batching efficiency, and model FLOPs utilization — always well below peak specs. Expected concurrency at peak, which determines how much total throughput the system must deliver. Utilization: the fraction of provisioned capacity actually used, because GPUs sitting idle cost the same as GPUs generating tokens. The effective cost per token is (GPU cost per hour) / (achievable tokens per hour × utilization). For the cost stabilization framework, see how to stabilize AI infrastructure cost.

The Estimation Method

Step 1: Model GPU memory — weight memory plus KV cache memory at target context length and concurrency. This gives the minimum GPU count. For the memory estimation method, see GPU memory for LLM inference. Step 2: Model throughput — per-GPU achievable tokens per second with batching, using realistic MFU, not peak. Step 3: Model concurrency — expected peak concurrent requests, which with throughput from Step 2 gives the serving GPU count. Step 4: Apply utilization — a realistic fraction (lower for latency-sensitive serving, higher for batch inference) to get provisioned GPU count. Step 5: Multiply GPUs by cost per GPU-hour to get serving cost estimate. Divide by expected tokens per month for cost per token.

For the GPU selection that affects all variables, see H100 vs A100. For batching that affects throughput, see LLM inference batching.

FAQ

How do I estimate LLM serving cost before deployment?

Model GPU memory (weights + KV cache at target context/concurrency), achievable throughput per GPU, expected peak concurrency, and realistic utilization. Multiply GPUs needed by cost per GPU-hour. Divide by expected tokens per month for cost per token. For the full method, see the steps above.

What is the biggest variable in LLM serving cost?

Concurrency and utilization. Even a model that fits on few GPUs can require many more at high concurrency with long contexts, because the KV cache memory per request adds up. Utilization — the gap between provisioned and used capacity — is where most cost estimates fail, because realistic utilization is often well below assumed levels. For the cost stability framework, see stabilize AI infrastructure cost.

Summary

Estimate LLM serving cost by modeling memory, throughput, concurrency, and utilization — a workload-first method producing cost per token. For the full cost framework, see how to reduce LLM inference cost.