A GPU Power Cap Taxes Training Harder Than Decode

A fleet-wide power cap is an easy policy and a bad model of the chip. The firmware holds the GPU under the watt limit by dropping clocks. Dense matrix multiplies — the body of a training step — light up enough of the die that the limit engages early. Decode for a large language model is often waiting on memory bandwidth, not on the next megahertz. The same 100 watts taken off both nodes is a different tax. Teams who publish one percentage for "the impact of power capping" have averaged away the only decision the percentage was for.

The Curve Is Not a Straight Line

Two observations from people who have measured this, stated as ranges because they are not a law of the H100.

• Near the top of the voltage-frequency curve, the next watt buys less clock than the previous one. Cutting power therefore costs less throughput than a linear reading of the wattage suggests. TechnoLynx's write-up of the governors makes this the point: the TDP is a ceiling the firmware enforces, not a dial that converts watts into FLOPs one for one.
• The shape of the tax depends on the kernel. Practitioner reports on H100-class GPUs put a drop from the 700 W default toward 600 W at roughly 5–8% throughput on bandwidth-bound inference, and higher, on the order of 12–15%, on compute-bound training GEMMs. Those are their ranges, on their stacks. They are a reason to measure yours, split by workload. They are not a number to put in a OneSource quote.

NVIDIA's own power-profile guidance says the same thing in operational language: apply a profile per workload class. A training job, an inference service, and an HPC kernel do not want one ceiling. If your cluster manager sets nvidia-smi -pl once per node image, you have implemented the opposite of that guidance.

Facility power decides whether you need the cap at all. A rack whose breakers cannot feed eight GPUs at the SKU limit will cap you whether or not the model wanted it. A rack built for that density lets the training nodes stay near the rated limit and lets you spend the tight cap where it is cheap. That is the facility side of OneSource Cloud: dedicated nodes in power envelopes sized for the GPUs, so a training cap is a choice rather than a condition of getting the machine turned on. The asymmetric tax, if you then choose a cap, does not go away.

Utilization Will Lie to You

A capped GPU is still executing. Data-center telemetry that reports "percent of time the GPU was busy" stays near the top while the clocks that decide FLOPs have already stepped down. The charts that move are SM clock and the throttle-reason field. A card sitting on its base clock instead of its boost clock can look fully utilized and deliver a step time you will blame on the dataloader, the network, or the batch size.

Read the clock next to the cap, per GPU, not as a node average. In a dense 8-GPU chassis the inner cards run hotter than the edge cards, and the firmware walks their clocks down further to hold junction temperature. A power cap and a thermal cap compound. Averaging the eight clocks hides the card that has become the straggler for every collective in the job.

The check

If power draw sits on the enforced limit for the length of a step, the cap is binding. Further cuts cost more than the last cut did. If clocks are down and power is nowhere near the limit, you are looking at thermals or a different governor, and a policy debate about watts is the wrong meeting.

Put the Tight Cap on the Decode Nodes

Split the policy the way the kernels split.

1. Leave training nodes at, or near, the SKU limit unless the breaker is the binding constraint. Measure step time when you do cap them. Do not infer the loss from the watt delta.
2. Cap decode nodes where you have confirmed the kernels are bandwidth-bound. Take the watts back as rack density or as heat the room does not have to remove. Re-measure time between tokens, not a FLOP counter.
3. Do not colocate a capped training job and an uncapped decode service on one chassis and hope the average is fair. The chassis has one thermal environment. The jobs do not have one governor.

None of these caps change tensor-core precision. The supported power interface throttles clocks inside the part's specification. Numerics stay. Step time and inter-token time are what you traded. A risk review that treats a power cap as a correctness change is reviewing the wrong object. A capacity review that ignores it is reviewing an incomplete one.

Questions worth answering before the policy is global

Does a power cap change training numerics?

No. The cap reduces clocks through NVIDIA's supported interface. It does not switch precision, and it does not perturb accumulators. What it changes is how long the step takes, and that change is larger on compute-bound training than on bandwidth-bound decode. If a run diverges after a cap, look at the usual causes — a failed rank, a bad checkpoint resume — not at the watt limit.

Does OneSource Cloud let a training job run uncapped?

OneSource Cloud sizes dedicated racks so a training node can sit near the GPU's rated power limit instead of a cap imposed by a shared breaker. You can still set a limit, and if you do, it still costs a training GEMM more throughput than it costs decode. The facility removes the reason to cap everything. It does not flatten the curve.