Measuring CUDA Graph Impact on LLM Inference Response Times

CUDA graph capture removes per-operation CPU launch overhead from GPU serving, and its effect on response times is a function of batch size, not a single number. Production serving stacks capture graphs across a configured set of batch sizes and replay them during decode, which is why honest measurement sweeps concurrency rather than sampling it. This page defines the costs that come with capture, gives the controlled procedure from eager baseline to captured sweep, and explains how to read decode gains, prefill neutrality, and variance correctly.

Prerequisites: What Graph Capture Changes and What It Costs

CUDA graph capture converts a sequence of GPU kernel launches into a replayable graph, removing the per-operation CPU launch overhead that dominates small-batch decode steps; in production serving stacks, the engine captures graphs for a configured set of batch sizes (commonly spanning roughly 1 to 256, mixing powers of two with intermediate sizes) and pads incoming batches up to the next captured size. Two costs come with the mechanism and belong in every measurement plan. Capture cost at engine start: initialization captures each configured size, which is why startup slows and why lazy-capture proposals exist in serving stacks. Memory cost at runtime: each captured graph holds workspace for its size, so widening the capture set trades GPU memory for coverage, which matters on nodes also holding KV cache. The prerequisite for honest measurement follows directly: the graph effect is not one number but a function of batch size, which the procedure must sweep rather than sample.

Serving-stack documentation makes the mechanism concrete: vLLM builds its CUDA graphs logic on piecewise compilation with dual runtime mode switching, and decode throughput in such stacks comes largely from graph replay with no Python execution inside a replay. Those design facts are what the measurement procedure isolates. They also explain why the procedure below emphasizes stability: the mechanism under test operates at the millisecond scale where the environment matters as much as the configuration, which is why teams run such sweeps on dedicated environments like OneSource Cloud single-tenant clusters.

• Mechanism: capture once, replay without per-op launch overhead
• Capture set: configurable batch sizes, commonly near 1 to 256
• Cost one: engine-start capture time
• Cost two: per-size graph memory on nodes shared with KV cache

The Measurement Procedure: Eager Baseline to Captured Sweep

Measure in six steps that isolate the graph variable. Step 1: freeze the configuration: same model, quantization, GPU, host, tensor-parallel degree, and memory settings for every run, because any drift contaminates the comparison. Step 2: run the eager baseline with graph execution disabled (in vLLM, enforce-eager mode), recording time to first token, per-output-token latency, and the p95 and p99 of both under a fixed request mix. Step 3: enable graph capture with the default size set, rerun the identical workload, and record the same four metrics. Step 4: sweep the load: repeat steps two and three at low, moderate, and high concurrency so the batch-size dependence becomes visible instead of sampled at one point. Step 5: record startup time and peak memory in both modes, capturing the two costs the latency numbers hide. Step 6: repeat each configuration at least three times and report medians with spread, not single runs, because launch-overhead effects sit inside millisecond-level noise that single samples misattribute. The design principle: the only variable that changes between compared runs is graph execution, and the output is a table over concurrency levels, not a sentence.

1. Freeze model, quantization, hardware, parallel degree, and memory settings.
2. Run the eager baseline, recording TTFT, per-output-token latency, and p95/p99.
3. Rerun the identical workload with graph capture enabled.
4. Sweep low, moderate, and high concurrency.
5. Record startup time and peak memory in both modes.
6. Repeat at least three times per configuration; report medians with spread.

The workload mix is the reader's choice, but it must be fixed across the comparison: sequence lengths, request rate, and prompt distribution all move TTFT and decode time on their own, which is exactly the contamination the frozen configuration prevents. Teams without a stable internal node can rent the stability instead: OneSource Cloud dedicated GPUs hold the hardware constant for the duration of the sweep.

Reading Results: Decode Gains, Prefill Neutrality, and Variance

Read the table against three expectations, then treat deviations as findings. Expectation one: per-output-token latency improves most at low-to-moderate batch sizes, because launch overhead is a large fraction of a small decode step and shrinks as batches grow toward compute-bound sizes. Expectation two: time to first token moves little, because prefill is dominated by compute rather than per-op launch, so large TTFT gains from graphs deserve a second look at what else changed. Expectation three: variance shrinks, not just the mean: graph replay removes jitter from the launch path, and published research on hybrid JIT-graph execution reports latency variance reduction as a distinct benefit, which p99 columns should reflect. Two verification checks close the procedure: confirm the padding effect, that per-token latency at a concurrency just above a captured size reflects work for the next size up, and confirm the memory ledger, that measured peak memory across the capture set stays inside the budget left for KV cache. Gains that survive this reading are real; gains that appear only in means at one concurrency level are sampling artifacts.

Environment stability is the silent validity condition. Millisecond-scale step times are exactly where neighbor load, frequency drift, and shared-tenancy scheduling add the noise that single runs misattribute to the graph effect. That is the argument for measuring on dedicated hardware: single-tenant bare-metal nodes hold topology, clocks, and memory budgets steady across the sweep, so eager-versus-captured differences reflect the mechanism rather than the neighbors. That determinism is a property of the OneSource Cloud delivery model, where single-tenant bare-metal clusters in secure U.S. data centers run one workload per node under flat-rate monthly pricing.

• Decode gains concentrate at low-to-moderate batch sizes
• Prefill and TTFT stay near neutral; large TTFT gains are a finding, not a win
• p95 and p99 should improve alongside the mean
• Check padding at concurrency just above each captured size
• Check the memory ledger against the KV cache budget

FAQ

Do CUDA graphs help prefill as much as decode?

No, not as the typical pattern. Prefill is compute-bound, so launch overhead is a small fraction of the step, while decode steps at small batch sizes are short enough that per-op launch overhead is material. Measurements therefore typically show per-output-token gains and near-neutral time to first token; a large TTFT gain usually means something else changed between runs.

Why do CUDA graph gains measure differently on dedicated GPUs than on shared clouds?

Graph-replay benefits live in millisecond-scale step times, so neighbor load, frequency drift, and shared-tenancy variance can swamp or fake the effect. Single-tenant bare-metal environments such as OneSource Cloud clusters hold topology, clocks, and memory steady across the sweep, so the eager-versus-captured difference reflects the mechanism rather than the neighbors, under flat-rate monthly pricing with zero egress fees.

Summary: A Table, Not a Sentence

The correct output of a CUDA graph evaluation is a concurrency-by-mode table with medians and tails, a startup and memory ledger beside it, and expectations applied in order: decode-heavy gains, prefill neutrality, variance reduction. Teams that measure this way decide capture-size sets from evidence; teams that sample once inherit defaults. OneSource Cloud dedicated environments provide the stable single-tenant hardware that makes the table trustworthy.

To benchmark serving configurations on dedicated single-tenant GPUs, explore OneSource Cloud private AI infrastructure.