Production Traffic Replay for GPU Capacity Sizing

Quick Answer: Production traffic replay sizes GPU serving by playing a recorded mix of real requests against a candidate pool. Synthetic QPS hides prompt length, tool calls, retries, and lunch-hour bursts. Replay makes those shapes visible before you buy another rack.

Production traffic replay is a capacity method that replays sanitized production traces onto a staging GPU pool so replica count and SKU choice follow observed demand, not a round number from a slide. It is a sizing lab, not a live shadow of current users.

This page is about sizing. It is not a scale-to-zero essay, not a failover-capacity plan, and not a shadow-traffic quality compare. Those pages assume you already know how many GPUs the mix needs.

What does a useful replay capture that QPS misses?

A request is not one token. Capture arrival time, prompt tokens, expected max tokens, model route, and whether the call was a retry. A 40 QPS lunch peak of 4k-token prompts is a different GPU problem than 40 QPS of 80-token classifications.

Replay the same hour on two candidate shapes: fewer large GPUs versus more small ones. Keep the trace identical. Change only the pool. That is the experiment. Changing the model and the pool at once tells you nothing about capacity.

How do you run a replay without leaking production data?

Sanitize first. Strip secrets, account identifiers, and raw documents. Keep token-length histograms and arrival timestamps. If a regulated team cannot release traces, synthesize a length-matched trace from those histograms. A length-matched synthetic is still better than “assume 512 tokens.”

Pin the model build, tokenizer, and decoding limits. Replay is a capacity instrument only if the work per request stays still. After a prompt or weights change, treat the old replay as stale. Run a new capture window.

Watch GPU time, queue delay, and rejected requests, not only average latency. A pool that meets a mean and drops 3% of bursts is undersized. Private AI infrastructure helps when you need an exclusive staging pool that will not steal production neighbors. It does not invent a trace you never captured.

When is replay the wrong first tool?

If you have no production yet, start with a scripted mix that names three prompt classes and a burst factor. Label it a hypothesis. Convert it to replay after the first real week. If you only have one SKU on order, replay still tells you replica count and max concurrency, not a fantasy shopping list.

OneSource Cloud’s exclusive U.S. halls, including Texas / Richardson, are useful when the replay pool must match production isolation. OnePlus Platform, OneSource Cloud’s AI orchestration platform, can hold the staging project next to production without mixing quotas. Neither replaces a week of honest traces.

FAQ

Is traffic replay the same as shadow traffic?

No. Shadow traffic copies live requests onto a candidate and compares answers or latency in parallel. Replay uses a stored window to size hardware. Shadow answers a quality or regression question. Replay answers how many GPUs that window needed.

How long should a sizing replay run?

Long enough to include one peak and one quiet period you actually serve. A 10-minute microbench will bless an undersized pool. A business week is better than a single Monday if your mix changes by weekday.

Can we replay on a smaller GPU and scale the result?

Only as a bound, and only if you keep sequence lengths equal. Do not assume a linear SKU conversion. Replay the candidate SKU for the peak hour you care about, even if the rest of the week stays on a cheaper estimate.

What if traces contain customer data we cannot copy?

Do not copy them. Publish length, arrival, and route histograms from the production account, then rebuild a synthetic replay in the lab. Capacity needs the shape. It does not need the customer’s sentences.

Summary

Production traffic replay sizes GPU serving from a recorded mix. QPS without length and arrival shape is a slogan. Sanitize traces, pin the model, and change one variable: the pool.

When the replay pool must stay off shared neighbors, review OneSource Cloud home and managed AI infrastructure for exclusive staging capacity rather than guessing from a synthetic QPS slide.