NUMA and PCIe Affinity Optimization for Enterprise LLM Serving Hosts

Most LLM serving performance tuning happens inside the engine, and most of the easy wins end there. Below the engine sits the host, and on the host the difference between a token generator that sprints and one that jogs is often NUMA and PCIe affinity: whether the process, its memory, and the network interrupts feeding each GPU sit on the NUMA node local to that GPU or hop sockets on every transfer. This guide walks the topology-first procedure: map the server, bind the serving stack, and verify the change with measurements that survive skepticism.

Prerequisites: Map the Topology Before Tuning Anything

Record three topology facts before changing anything: the GPU-to-CPU affinity matrix from nvidia-smi topo -m, the NUMA node CPU and memory ranges from numactl, and the PCIe ancestry of each GPU to find its root complex and attached NICs. Every binding decision later derives from this map, and a host whose map is undocumented cannot be tuned repeatably.

Three discovery passes give you the map every later decision derives from:

1. Run nvidia-smi topo -m and save the matrix: it shows GPU-to-GPU connectivity (NVLink, PCIe, PIX/PHB) and the CPU affinity list for each GPU.
2. Run numactl -H to record each NUMA node, its CPU range, and its memory capacity.
3. Trace PCIe ancestry for each GPU and its attached NIC (lspci tree) so you know which NIC shares a root complex with which GPU.

Save the output per server SKU and date it. A host whose topology map is undocumented cannot be tuned repeatably, and drift checks later compare against exactly this record.

Binding the Serving Stack to the Local NUMA Node

Bind the serving process and its host-memory allocations to the NUMA node local to the GPU it drives, using numactl or the launcher environment for the process and memory policy, and steer the interrupt queues of the NIC feeding that GPU toward the same node. The goal is that weight loads, activation transfers, and network arrivals for each GPU cross the minimum distance on the PCIe and interconnect path instead of hopping sockets.

Apply the bindings in this order, one GPU group at a time:

1. Partition GPUs into node-local groups from the topology map: each group contains the GPUs whose local CPUs and memory sit on one NUMA node.
2. Bind the serving process for each group to that node with numactl (or the launcher environment), covering both CPU affinity and memory policy so allocations stay node-local.
3. Steer NIC interrupts for the NIC feeding each group toward the same node, so network arrivals for a GPU land on the CPUs that can hand them to the GPU fastest.
4. Persist the settings in the launcher or systemd unit rather than in shell history, so bindings survive restarts and reboots.

The goal is mechanical: weight loads, activation transfers, and network arrivals for each GPU should cross the minimum distance on the PCIe and interconnect path. On multi-GPU hosts whose GPUs span both nodes, per-GPU processes must be split accordingly; binding everything to one node is a regression there, not an optimization. Platform-level placement complements these host bindings: at fleet scale, the OnePlus AI Orchestration Platform places communication-intensive workloads on the nearest topology units when scheduling, which is the same locality logic applied one layer up.

Verification: Proving the Change Helps and Keeps Holding

Measure before and after with the same replayed load, comparing tokens per second, p99 time to first token, and host-side metrics such as cross-socket traffic; accept the change only when the after-run improves beyond run-to-run variance. Schedule drift detection by re-checking bindings after driver updates, reboots, and supervisor changes, and alert when a serving process is found running off its assigned node.

1. Measure before and after with the same replayed load: tokens per second, p99 time to first token, and host metrics such as cross-socket traffic or NUMA remote access counters.
2. Accept the change only when the after-run improves by more than your measured run-to-run variance; smaller deltas are noise wearing a costume.
3. Wire drift detection: after every driver update, reboot, or supervisor change, re-check that serving processes still sit on their assigned nodes, and alert when one is found elsewhere.

One structural advantage is worth stating: affinity settings only hold when host topology is stable and fully visible. Single-tenant bare-metal servers, such as those OneSource Cloud provides in secure U.S. data centers, expose the physical NUMA and PCIe hierarchy directly to the tenant with no hypervisor CPU scheduling or migration layer shifting bindings between runs, so the tuning you apply is the tuning that keeps working.

FAQ

Does NUMA pinning help single-GPU inference or only multi-GPU servers?

Both, for different reasons. A single-GPU host gains by keeping the serving process, its memory, and its NIC interrupts on the node local to that GPU. Multi-GPU hosts gain more when each GPU group is placed on its own node, provided the serving processes are split per group; binding everything to one node on a dual-socket host is a regression, not an optimization.

Why does bare-metal capacity matter for host-level serving tuning?

Because tuning assumes a stable, fully visible host. OneSource provides single-tenant bare-metal GPU servers, so the NUMA and PCIe topology you tune is the topology you own, with no hypervisor CPU scheduling or migration layer silently moving your bindings between runs.