How to Test Noisy-Neighbor GPU Latency Before Production

A noisy-neighbor GPU latency test is a controlled load that runs your serving path while another job contends for the same device, node, or fabric, so you can see p99 move before production does. If you only benchmark an idle exclusive GPU, you have not tested the tenancy you will actually buy.

The test has three numbers: a quiet baseline, a contended run, and the delta on the percentile you sell. Teams that skip the contended run discover jitter as a customer incident. The rest of this article is a procedure you can run in a staging account or on a private node.

What you are actually testing

Noisy neighbor is not one failure. It can be SM time-slicing on one GPU, HBM or PCIe contention on one node, or an all-reduce saturating the same NIC your tokens use. A useful test names the shared layer. “The GPU is slow” is not a layer.

A procedure that fits in one afternoon

Pin your serving replica to known GPUs. Record p50 and p99 on a replay trace for ten minutes. Start the contender on the layer you claim to share. Repeat the trace. Stop. If p99 moves outside the SLO margin, that tenancy is not a production serving pool. Do not average the two runs into one “GPU latency” slide.

Hold input tokens, batch size, and model constant. Change only the neighbor. If you also change instance family, you learned nothing about tenancy. Capture nvidia-smi, DCGM, and fabric counters for both runs so the ticket is not an argument about feelings.

How to read the result

A small delta on exclusive, single-tenant GPUs is the expected shape. A large delta on shared cloud GPUs is also expected; it is why exclusive capacity exists. MIG can isolate small jobs on one device and still fail the node or NIC test. Publish the layer you tested. A passing GPU test with a failing fabric test is a fail for online serving.

If the business cannot accept the delta, move serving off shared tenancy. Private AI infrastructure at OneSource Cloud is exclusive GPU capacity where this test should be boring. Keep serving and training on separate queues with OnePlus, OneSource Cloud’s AI orchestration platform, so a neighbor is a policy mistake rather than a default. Fabric contention belongs on AI networking, not only on the device. Managed operations keep the same counters in production so the staging test is not a one-off.

FAQ

How do you test noisy-neighbor GPU latency before production?

Take a quiet baseline on a pinned serving replica, start a contender on the shared GPU, node, or fabric you actually bought, and compare p99 on the same trace. The test fails if the percentile you sell moves outside the SLO margin. Idle-only latency numbers are not this test.

Is MIG enough to pass a noisy-neighbor test?

It can pass a same-GPU isolation test for small inference. It does not automatically pass node PCIe or NIC tests. If training all-reduce shares the host, serving p99 can still jump. Test the layer your architecture actually shares, not the marketing name of the partition feature.

What percentile should we watch?

Watch the percentile in the product SLO, usually p95 or p99, not the mean. Means hide the tail that pages on-call. If you have no SLO yet, publish p99 on a fixed replay before you compare tenancy options. Changing traces between runs invalidates the delta.

Should training and inference share a GPU during the test?

Only if they will share one in production. Testing a neighbor you will never run is theater. If production serving is reserved exclusive, the relevant test is another serving replica or a maintenance job on the same node, not a 64-GPU training sweep on a different cluster.

When should we stop using shared GPU cloud for serving?

When the contended p99 cannot meet the SLO after you have pinned placement and still share tenancy. At that point exclusive GPUs are a latency control, not a brand preference. Keep shared tenancy for batch that can wait.

Summary

Test noisy-neighbor GPU latency with a baseline, a named contender, and p99 on the serving path you will sell. Shared tenancy that fails this test is not a production inference pool. If exclusive GPUs are how you keep the delta boring, evaluate OneSource Cloud private AI infrastructure and keep serving off the training queue on OnePlus.