Put the GPU Failure Domain in the Contract, Not the SLA

Shared-responsibility diagrams are pyramids. A dead GPU is four verbs. Someone notices, someone drains the node so new work stops landing on it, someone replaces the part, and someone decides the node may return to the scheduler. An SLA percentage names none of those verbs. A credit paid after the fact does not load your checkpoint. If the contract you are about to sign cannot assign an owner to each verb, the percentage is theater.

Four Actions, and Most Contracts Name One

Classic shared-responsibility language, the kind AWS publishes and everyone else roughly copies, gives the physical host to the provider and the guest workload to the customer. That split is necessary and it is not sufficient. The operational handoff — the hour between "the GPU is sick" and "the GPU is back, and we trust it" — is what the pyramid leaves to a contract that often never writes it down.

NVIDIA's Xid messages are the documented hardware-side signal for a large class of these faults. Cite them in the detect row. "The provider monitors the infrastructure" is not a signal, and it cannot be audited after an incident. Neither is a promise of zero failures. GPUs fail. The contract's job is to say whose pager fires.

On a managed dedicated cluster the host rows sit with the operator of the machine. OneSource Cloud runs the dedicated hosts: the drain and the hardware replacement are provider work, on hardware that is not shared with another tenant while it is being repaired. That is a cleaner RACI than a multi-tenant node, where "drain" also means deciding whose other jobs were on the box. It is not a transfer of the workload. Your job, your checkpoint, and your rule for trusting the node again stay on your side of the table.

A Credit Does Not Resume the Job

Even after the host rows are clearly the provider's, three decisions remain yours, and an SLA will not make them.

• What the checkpoint contains. A provider can hand you a fresh GPU and a stale file. If that file is weights without optimizer state and without the data-loader position, the resumed run is a different run. That bug is not hardware.
• How many times you retry. A blind restart loop will put the job back on a flapping node and call it resilience. Cap the retries. Alert on the count.
• When a node is trusted. Replacement is not a health check. A short burn-in — a known-answer GEMM, an all-reduce against its neighbors — belongs in the return row, and someone has to own the decision to skip it. Skipping it should be a named exception, not the default because the launch is late.

Write those three next to the provider's three. A RACI with an empty customer column is how teams discover, during the incident, that "managed" did not include the retry policy they assumed it included.

Ask for the Record, Not the Percentage

The exhibit that matters after an incident is short.

1. The Xid or other signal, with the timestamp and the GPU id.
2. The time the node was cordoned, and who cordoned it.
3. The time a replacement was in the chassis and past the firmware check.
4. The check that had to pass before the scheduler could see the node, and the result.

Ask for these in the service description. Do not assume a provider already retains all four because a marketing page says the platform is observable. If the answer is "we can add that," write the addition into the exhibit before signature. A percentage point of availability, negotiated hard, does not reconstruct this timeline. The timeline is what tells you whether the failure domain in the contract matches the failure domain you lived through.

What the Contract Still Cannot Promise

Silent corruption that does not throw an Xid. A checkpoint written after the fault and then treated as truth. A job whose watchdog is shorter than its own checkpoint barrier, so healthy runs get killed. Assigning owners does not delete those. It stops them from being nobody's. A risk committee that wants a remaining-risk statement can have this list. It should not be given a fake zero.

Questions reviewers ask too late

Does a BAA decide who replaces the GPU?

No. A business associate agreement covers how protected health information is handled. It does not assign detect, drain, replace, or return. Those rows belong in the service description or an operations exhibit. A review that treats the BAA as the infrastructure RACI has not reviewed the infrastructure.

Which GPU failures does OneSource Cloud own?

OneSource Cloud operates the dedicated host. Detection support, the drain, and hardware replacement sit with the provider, on hardware that stays single-tenant through the repair. The training job's checkpoint, its retry limit, and the rule that keeps a suspect node out of the scheduler stay with the customer. A credit is not one of those rows, and it does not resume the job.