LLM Infrastructure Deployment: Serving Stack and GPU Sizing

LLM infrastructure deployment is the process of putting a large language model into production on GPU servers, together with the inference engine, API gateway, autoscaling, and monitoring needed to serve requests at a defined latency and cost. The model file is the smallest part of the job. Most deployment problems come from three calculations teams skip: how much GPU memory the KV cache needs at real concurrency, how many replicas a latency target requires, and which quantization level keeps accuracy acceptable.

The Layers of an LLM Serving Stack

A production deployment has more parts than a demo. Each layer has a decision and a common failure.

Choosing an Inference Engine

The inference engine determines throughput per GPU more than any other software choice. The main open options take different approaches.

Benchmark at least two engines with your model, your prompt lengths, and your concurrency. Published benchmarks rarely match a specific workload, and the gap between engines changes from release to release.

Sizing GPU Memory: Weights Plus KV Cache

GPU memory for inference has two parts. The weights are fixed. The KV cache grows with every token of every active request:

KV cache per token = 2 × layers × KV heads × head dimension × bytes per value

For Llama 3.1 70B (80 layers, 8 KV heads, head dimension 128) in BF16, that is 2 × 80 × 8 × 128 × 2 bytes, about 0.33 MB per token. One request holding a full 8,192-token context therefore uses about 2.7 GB of KV cache.

Worked Example: How Many Concurrent Requests Fit

Take four 80 GB H100 GPUs serving Llama 3.1 70B in BF16 with tensor parallelism:

Real requests rarely use the full context, so the engine can usually hold more active requests than this worst case. The calculation still sets a safe planning floor and shows why long-context features are expensive: doubling the context to 16,384 tokens halves the worst-case concurrency.

Quantizing weights to FP8 frees about 70 GB, and FP8 KV cache halves the per-token cost. On Hopper and Blackwell GPUs, FP8 is supported natively, which is why it is a common production choice for large models.

Latency and Throughput: The Metrics That Matter

Users experience two latencies, and they respond to different changes.

Total response time is roughly TTFT + TPOT × output tokens. With a TTFT of 0.4 seconds, a TPOT of 30 ms, and a 500-token answer, the full response takes about 15.4 seconds, though streaming lets users start reading after 0.4 seconds. Raising batch size increases throughput but also raises TPOT, so every deployment sets a target for each and measures goodput rather than raw throughput.

Capacity Planning: From Traffic to Replicas

Turn expected traffic into a replica count with three numbers: peak requests per second, average output tokens per request, and measured goodput per replica at your latency target.

Replicas = (peak requests per second × average output tokens) ÷ goodput per replica, rounded up, plus redundancy

For example, 20 requests per second × 400 output tokens needs 8,000 output tokens per second. If load testing shows one 4-GPU replica sustains 2,500 output tokens per second while meeting the TTFT and TPOT targets (an illustrative figure; measure your own), you need 3.2 replicas. Rounding up gives 4, and adding one for failover and rolling updates gives 5 replicas, or 20 GPUs.

Two refinements change the result in practice. Prompt-heavy workloads such as RAG with long retrieved context are limited by prefill compute, so measure TTFT under load, not only output throughput. And workloads where many requests share a long system prompt benefit strongly from prefix caching, which can raise goodput without more GPUs.

Quantization Choices

Quantization is an accuracy decision as much as a cost decision. Run the quantized model against the same evaluation set used to approve the full-precision model, and record the results before it goes to production.

Rollout Plan With Pass Criteria

Model weights for large LLMs take time to load onto GPUs, so new replicas are not instant. Keep weights on fast local or nearby storage, and set scaling thresholds early enough to absorb load time.

Where to Run It: API, Cloud GPUs, or Dedicated Infrastructure

The break-even between per-token APIs and dedicated GPUs depends on sustained volume. Cost per million output tokens on your own GPUs is GPU cost per hour × GPUs per replica ÷ (goodput in tokens per second × 3,600) × 1,000,000. Compare that figure with API pricing at your real monthly volume. OneSource Cloud's analysis of inference economics at scale covers this comparison in more depth.

OneSource Cloud deploys LLM serving on dedicated AI GPU Clusters in U.S. data centers, including Richardson, Texas. The OnePlus Platform, OneSource Cloud's AI orchestration platform, provides inference services, Kubernetes lifecycle management, and developer tools on that infrastructure. See the OnePlus Platform.

FAQ

How many GPUs do I need to deploy a 70B LLM?

In BF16, weights alone need about 140 GB, so at least two 80 GB GPUs, and four for meaningful concurrency at long context. With FP8 weights and KV cache, two 80 GB GPUs can serve the model with moderate concurrency.

Which is better for production, vLLM or TensorRT-LLM?

vLLM is easier to adopt and supports more models out of the box. TensorRT-LLM often delivers higher throughput on NVIDIA GPUs for a fixed set of models but requires building engines per model and GPU type. Benchmark both with your workload before deciding.

What should LLM inference autoscale on?

Scale on signals tied to GPU saturation: request queue depth, KV cache utilization, or in-flight requests per replica. CPU utilization is a poor signal because the GPU is the bottleneck.

What is a good time to first token for an enterprise chatbot?

Many teams target well under one second at p95 for interactive chat, with streaming output. Long-context RAG requests have higher TTFT because the whole prompt must be processed first, so set separate targets per request type.

How long does it take to deploy an LLM on private infrastructure?

With GPUs already available, a single model can be benchmarked, load-tested, and released in stages within weeks. Most of the time goes to evaluation, load testing, and security review rather than installation.

Summary

Deploying LLM infrastructure is a sizing and validation exercise. Size GPU memory for weights plus KV cache at real concurrency, using about 0.33 MB per token for a 70B model in BF16. Turn traffic into replicas with measured goodput at your TTFT and TPOT targets. Choose quantization against your own evaluation set, autoscale on GPU-saturation signals, and roll out in stages with pass criteria at each step. Steady, sensitive, or proprietary workloads are usually the strongest case for dedicated infrastructure.

To size an LLM deployment on dedicated GPUs, explore OneSource Cloud's AI GPU Cluster service.