GPU Xid Error Meaning for Enterprise AI Operations

Quick Answer: A GPU Xid is an NVIDIA kernel-reported event about the device, the driver, or the board path, not a Python traceback. Read the code, the count, and whether the GPU reset, then decide drain, driver rollback, or hardware replacement. Do not treat every Xid as an application bug, and do not ignore a repeating Xid 79 as “just training.”

A GPU Xid error is a numeric event from the NVIDIA driver that records a device-side fault, reset, or memory exception during AI operations. It is a hardware and driver signal. It is not a framework log line and not a full postmortem by itself.

Cluster operators should wire Xid collection before the next burned job. This page explains what the signal means. It is not a generic “GPU cluster troubleshooting” playbook and not an ECC-only procedure.

What does an Xid actually tell you?

Xid numbers are documented by NVIDIA and show up in dmesg and persistence-daemon logs. Common ones operators meet: Xid 13 or 31 for illegal or timeout-class GPU work, Xid 48 for uncorrectable ECC, Xid 79 when the GPU falls off the bus. Do not memorize a poster. Record the code next to the PCI address and the job id.

How should on-call classify Xid events?

Classify first as board, driver, or workload. Board means the same GPU keeps faulting across jobs and after a clean reboot. Driver means a fleet-wide change moved the rate. Workload means one binary, one CUDA version, or one illegal kernel reproduces on healthy neighbors.

Then classify severity by user impact. A reset that kills every replica on the node is a page. A single correctable-adjacent warning that did not reset can wait for the next change window if counters stay flat. Repeating Xid 79 after reboot is an RMA conversation, not a PyTorch issue.

Shared GPU clouds hide the PCI address and the neighbor. You cannot classify what you cannot see. Exclusive cards on private AI infrastructure make the address, the power shelf, and the job unique. That is why Xid handling is an isolation problem as much as a decoder problem.

Which actions are safe after an Xid?

Drain the node if the GPU reset or vanished. Do not reschedule training onto a device that just fell off the bus so you can “use the hours.” Capture nvidia-smi -q, dmesg, Xid line, firmware versions, and the last driver package. If you run MIG, note the instance ids. Lose that bundle and the vendor will bounce you.

Rollback the last driver or fabric-manager change if the rate jumped fleet-wide. Xid storms after an upgrade are common enough that a canary node should exist. If you have no canary, the first production node is the canary. That is an operations defect.

Rerun the same job on a neighbor GPU before you rewrite the model. If the neighbor is clean, stop blaming the checkpoint. If the neighbor fails the same way, you have a workload or image problem. Managed AI infrastructure is useful when someone already has the drain and capture runbook at 02:00.

What Xid will not explain?

It will not explain a logical NaN, a bad tokenizer, or a silent quality drop. Those are application traces. It will not replace ECC counters, NVLink error counters, or PDU logs. Xid 79 especially wants power and PCIe link evidence, not another learning-rate theory.

It will not, by itself, prove a neighbor attacked you. A reset on a shared host can be their kernel or your kernel. If you need forensic isolation, you needed exclusive GPUs before the incident. OneSource Cloud keeps dedicated devices in U.S. facilities, including Texas / Richardson, so an Xid maps to one customer-facing board.

OnePlus Platform, OneSource Cloud's AI orchestration platform, should attach the Xid, hostname, and GPU index to the failed job. A dashboard that only says “job failed” forces a hunt through syslog. That hunt is how outages grow.

FAQ

Is every Xid a broken GPU?

No. Some codes follow an application that issued illegal work. Some follow a driver bug. A repeating code on one board after reboot and after a different image is when you treat the GPU as failed. One isolated Xid 13 on a brand-new job is usually the job.

What does Xid 79 usually mean?

Operators treat “fallen off the bus” as a device or path failure until power, PCIe, and firmware say otherwise. Drain the node. Do not keep the GPU in the serving pool. Collect the board serial before anyone reseats blindly and loses the evidence.

What does Xid 48 mean?

It is in the uncorrectable ECC family. Switch to the ECC handling procedure: check retired pages, double-bit counts, and whether the device should leave the pool. Do not debug CUDA kernels first. Memory is the story until counters disagree.

Should we page on the first Xid?

Page on reset, disappearance, or a rising rate. A single non-reset event can be a ticket. Write the threshold in the runbook so night shift does not invent it. Serving pools should be stricter than research queues.

Do we need vendor attestation to trust Xid logs?

No. Xid is a driver log, not a compliance certificate. Keep it in your operations evidence. It does not prove isolation or replace a SOC 2. For research labs that share clusters, see AI for research scheduling notes only if multi-lab access is the actual design.

Summary

Read an Xid as a device-side event: note the code, the PCI address, the reset, and the rate. Classify board versus driver versus workload before you RMA or rewrite training. Drain on reset. Rollback on fleet-wide jumps. Capture firmware before anyone reseats the card.

Exclusive U.S. GPUs make that classification possible. Shared hosts scramble it. If you want Xid lines tied to jobs you own, run them on dedicated devices and keep the syslog path in the operations contract, including OneSource Cloud managed operations when you do not staff nights.