LLM Inference Failover Capacity Planning for Production

Quick Answer: LLM inference failover capacity is the extra serving GPUs you keep so a replica, node, or site can disappear and the remaining pool still meets the SLO. Do not spend that spare on a Tuesday deploy. That spend belongs to headroom.

Inference failover capacity is reserved serving compute that absorbs a named failure domain so production traffic continues without recycling ordinary headroom or untested training cards. The plan names the domain, the spare, and the rehearsal. A diagram without GPUs is not a plan.

This page sizes the spare. It is not a private-cloud isolation-boundary article and not a daily headroom calculator. Isolation keeps tenants apart. Headroom covers deploys. Failover covers loss.

Which failure domain are you actually sizing?

Pick one primary domain for the first plan. Most chat fleets start at node loss. Site loss is a residency and DNS decision, not a percent you invent on Friday. Private AI infrastructure can keep both sites exclusive. It does not copy weights for you.

How do you turn the domain into GPU count?

Take useful peak from the serving histogram, the same peak used for headroom. Subtract the GPUs inside the failure domain. The remainder must still serve that peak. The missing GPUs are failover capacity. They must be imaged, warmed, and on the same model revision, or the recovery time must be written as an accepted degrade.

N+1 means one extra node (or replica) above the minimum that holds peak. A percent means a fraction of the pool. N+1 is clearer when nodes are large. A percent is clearer when you have many small replicas. Do not mix them in one spreadsheet without labeling the unit.

Long-context and high-concurrency shapes need more leftover KV and batch room on the survivors. If survivors go over memory, failover fails even though GPU count looks fine. Size memory shape, not only card count.

What must not sit in the failover column?

Ordinary deploy headroom is not failover. If a release is in progress and a node dies, you need both numbers. Training burst cards are not failover until they have served production traffic in a rehearsal. A second model that cannot load the production tokenizer is not failover.

Isolation rules still apply. Failover replicas that land on a shared neighbor can change tail latency even if they restore QPS. If the program requires exclusive GPUs, the spare must be exclusive too. AI networking and storage paths on the spare must match the primary, or the rehearsal will fail on data movement rather than on compute.

U.S. data-residency programs, including Texas / Richardson halls, often forbid failing over into an unnamed region. Write that constraint as a reduce-traffic degrade, not as a silent cross-region hop. OneSource Cloud does not publish a multi-region SLA on this page.

How do you know the spare works?

Rehearse the named domain. Drain a node in a change window and watch SLO, queue time, and memory on survivors. If product traffic cannot be used, replay a frozen trace at useful peak. A failover you have never pulled is inventory.

Managed AI infrastructure can run the drain and the watch. Someone still signs that the spare stayed unused during ordinary deploys. If the last incident spent failover on a rollout, the plan already failed.

FAQ

Is failover capacity the same as inference headroom?

No. Headroom is the buffer you expect to use for deploys and jitter. Failover capacity is the buffer you expect to keep full until a named domain fails. One number cannot do both jobs on the same afternoon.

Do we need a second site to claim failover?

Only if the named domain is the site. Node-level failover can live in one hall. Site-level failover needs a second loaded location or an accepted outage. Do not call DNS a capacity plan.

Can we keep failover GPUs powered off?

Only if the time to image, load weights, and pass a smoke test still meets the SLO. For most chat fleets that window is longer than the error budget. Warm spare is the default. Cold spare is a documented degrade.

Does dedicated GPU tenancy change the math?

It changes who else is on the survivor nodes. It does not change useful peak. Exclusive spare is easier to reason about. You still subtract the failed domain and count remaining GPUs.

What should we write when residency blocks a second region?

Write a degrade: lower concurrency, longer queues, or a read-only mode inside the allowed boundary. Do not pretend a blocked hop is a failover. Keep the legal boundary and the capacity story on the same page.

Summary

LLM inference failover capacity is warm serving spare sized to a named failure domain. It is not daily headroom and not untested training stock. Survivors must hold useful peak, including memory shape.

Rehearse the drain. Explore OneSource Cloud’s private AI infrastructure when both the primary pool and the spare must stay exclusive and inside a named U.S. boundary.