How to Size GPUs for Embedding Backfills in Enterprise Data

Sizing GPUs for embedding backfills means converting corpus tokens, chunk size, and a deadline into a batch GPU reservation that can finish a re-embed without taking production inference cards. A policy change, a model swap, or a deletion rebuild is not “a bit of extra embedding.” It is a timed batch job with a known token volume.

Teams undersize this when they use the online embedding replica count as the backfill fleet. Online replicas are sized for query bursts. Backfills are sized for a mountain of chunks. Mixing them is how search quality work becomes an SLO incident.

Inputs you actually need

Run a timed sample of 10k chunks on one GPU with the real model and the real storage path. Extrapolate. Do not extrapolate from a laptop. Chunker CPU and object-store latency are part of the rate. GPU-only math will over-promise.

Reservation, not leftover fair share

Put backfills on a batch partition. Cap them so they cannot drain serving. If the deadline is hard, buy or reserve a temporary extra slice rather than preempting chat. A corpus change is usually planned; treat it like a maintenance window. Unplanned legal deletes may still need a drain plan that serving can survive.

Checkpoint embedding progress by document ID so a preemption does not restart the mountain. The same ID discipline that deletion needs is what makes backfill restartable. If IDs are paths that change, you will double-embed and still miss documents.

Where the job should run

Backfills read the whole corpus. They should run next to that corpus, not across a public embedding API that creates a second copy. Exclusive GPUs plus private storage keep the rebuild inside one residency boundary. That matters more when the corpus is regulated than when it is a blog mirror.

OneSource Cloud’s AI storage architecture is the read path. Batch GPUs sit on private AI infrastructure. OnePlus, OneSource Cloud’s AI orchestration platform, is how the backfill queue stays off the serving reservation. If the corpus is clinical, stay on healthcare AI infrastructure and still isolate the job. HIPAA-ready is about the environment. Throughput is about the reservation.

FAQ

How do you size GPUs for an embedding backfill?

Measure chunks per second on one GPU with the production model and storage, divide remaining chunks by that rate, then divide by the hours in the deadline. Add headroom for retries and I/O stalls. If storage cannot feed the GPUs, add disks or readers, not only cards. Publish the sample method with the reservation request.

Should online embedding replicas do the backfill?

No, unless the backfill is tiny and you have SLO margin. Online replicas exist for query latency. A full corpus rebuild will queue or slow those queries. Use a batch partition with its own cap. Merge results into the index in a controlled cutover, not by hammering the serving embed path.

What if we change chunking and the embedding model at once?

Treat it as a new index, not an in-place update. Size GPUs for a full rebuild. Keep the old index serving until the new one passes retrieval tests. Dual-write periods need storage and GPU for both. Trying to save capacity here is how you serve a half-migrated schema.

Can we use leftover training GPUs overnight?

Yes if reclaim returns them before morning inference, and if the backfill checkpoints. No if training overruns or if serving needs those cards as failure margin. Overnight borrow is a written exception. It is not the default capacity plan for a legal deletion deadline.

Does embedding backfill need the same GPUs as LLM serving?

Not always. Many embedding models fit on smaller GPUs than the chat model. Mixing them on one SKU wastes HBM or underpowers the LLM. Size the backfill SKU to the embedder. Keep the big cards for serving and training unless you only have one family.

Summary

Size embedding backfills from measured chunk rate, corpus size, and deadline, then isolate them from inference. Storage stall is a first-class input. If the rebuild must stay on exclusive U.S. GPUs next to the corpus, use OneSource Cloud storage with private AI infrastructure and queue the job on OnePlus.