RTO and RPO Requirements for AI Workload Recovery

RTO is the time you can afford to be down. RPO is the amount of recent work you can afford to lose. For AI systems those two numbers cannot be one pair for the whole stack. A serving endpoint, a training checkpoint, and a vector index restore on different clocks and fail in different ways.

Teams that copy a 4-hour RTO from an ERP runbook discover that GPU weight loads, index rebuilds, and in-flight training steps do not honor it. The fix is to assign objectives per artifact, then design backups and failover to those objectives.

This article maps the artifacts, shows why one RPO is a lie, and lists the evidence a recovery test must produce.

AI Artifacts Have Different Clocks

KV cache and in-flight batches are generally not recoverable. If the node dies, those tokens are gone. Do not write an RPO of zero for decode state unless you have a design that persists it, which most stacks do not.

Set Objectives From Business Impact, Then From Physics

Start with the user-visible failure. If the assistant is down, what revenue or clinical workflow stops, and in how many minutes does that become unacceptable? That sets serving RTO. If a training run dies, how many GPU-hours can you replay? That sets checkpoint RPO.

Then apply physics. A 70-billion-parameter model does not become healthy in 30 seconds if weights must cross a slow link onto cold GPUs. A vector index that takes six hours to rebuild cannot support a 15-minute RTO unless you keep a hot replica. When the business number and the physics number disagree, change the architecture or change the commitment. Do not sign the smaller number.

Write both numbers with scope. "RTO 30 minutes for the production chat endpoint in region X" is a requirement. "RTO 30 minutes for AI" is a slogan.

Design Backups to the Tightest Honest RPO

Checkpoint RPO is the interval plus the risk that the last checkpoint is corrupt. If you checkpoint every two hours and never validate the file, your real RPO is worse than two hours. Add a checksum and a restore smoke test to the interval, or admit the gap.

Index RPO depends on whether you can replay from object storage. If documents remain and the index is disposable, RPO follows the document store, and RTO follows rebuild time. If the index is the only copy of chunk text, you just created a second system of record. Back it like one.

Store recovery copies in a location that still meets residency rules. A backup that leaves the approved region solves availability by creating a compliance incident. Put that constraint in the AI storage architecture design, not in a ticket after the first drill.

Failover Paths That Match the RTO

Serving RTO is usually met with a warm standby or a second replica that can take traffic, not with a restore from object storage. Loading weights is faster when GPUs are already up and the image is local. That is a capacity cost. Pay it if the RTO requires it; do not pretend a cold cluster meets a short clock.

Training RTO is different. You rarely need the same job back in minutes. You need the last good checkpoint and a cluster that can be reassembled. Dedicated capacity helps because the hardware inventory does not vanish into a regional GPU shortage during the same event that interrupted you.

On private AI infrastructure, define whether failover stays inside one hall, moves to a second U.S. site, or waits on repair. OneSource Cloud treats that as a capacity and runbook problem, not a slogan. Managed AI infrastructure should include the restore procedure, the contact path, and the last successful test date. An untested RTO is a wish.

Evidence a Recovery Test Must Leave Behind

A drill is complete only when you have timestamps and artifacts:

• Start and end times for detection, failover, and full service restoration.
• The exact checkpoint or image that came back, with checksums.
• Data loss window measured from the last durable write, not estimated.
• Residual errors after traffic returned, including failed ranks or stale indexes.
• A decision to accept the numbers or to change architecture or the stated RTO/RPO.

Run the drill on a copy of production topology, or on production during a planned window you can afford to fail. Tabletop reviews do not measure weight-load time.

FAQ

What RTO and RPO should we set for AI workloads?

Set them per artifact. Serving often needs a short RTO and an RPO of the last released model. Training often accepts a longer RTO and an RPO equal to the last validated checkpoint. Indexes need a pair that matches rebuild time or replica lag.

Can we claim zero RPO for LLM inference?

You can claim zero loss of released weights if those artifacts are already in a registry. You cannot honestly claim zero loss of in-flight generations or KV cache after a node crash. Separate those statements.

How often should we checkpoint training?

Often enough that lost GPU-hours stay acceptable, and rarely enough that checkpoint I/O does not dominate step time. The interval is an RPO decision with a performance cost. Measure both.

Does dedicated GPU cloud improve RTO?

It can, because reserved GPUs and local images remove marketplace wait time. It does not help if you never tested restore, or if the only copy of the index sits on the failed node.

Who owns AI recovery objectives?

The service owner sets the business numbers. Platform and storage owners confirm they are physically possible. If those groups do not meet, the signed RTO is fiction.

Summary

RTO and RPO for AI only work when they are attached to weights, checkpoints, indexes, and logs separately. Derive the numbers from user impact, then constrain them by restore physics. Design replicas for short serving clocks, validated checkpoints for training, and residency-safe backups for everything durable. Test with timestamps, or do not claim the objective.

OneSource Cloud can help map recovery objectives onto dedicated U.S. GPU capacity and operations runbooks. Request an architecture review to turn a single slogan RTO into per-artifact numbers you can test.