CPU vs GPU for Enterprise LLM Inference Workloads

Quick Verdict: Keep embeddings, rerankers, and small classifiers on CPU when latency budgets and batch shapes allow it. Put autoregressive decode for large language models on exclusive GPUs. Do not park a 70B chat replica on CPU because a blog said quantization is magic, and do not burn an H100 on a 100-million-parameter encoder.

CPU vs GPU for inference is a placement decision about which stage of the request needs dense matrix throughput. CPUs win on sequential control, wide fan-out of tiny models, and memory-capacity jobs that do not stream tokens. GPUs win when each request walks a large weight set many times per second.

Platform owners should split the graph before they split the purchase order. This page is an inference-placement guide. It is not a GPU memory calculator and not a training-hardware comparison.

How should CPU and GPU split an enterprise inference graph?

A RAG request is usually both. Retrieval and chunk scoring can stay on CPU. The generator that writes the answer usually cannot. Design the path as two pools with two SLOs, not one “AI server” that hides the split.

When is CPU the right inference host?

CPU fits when the model is small enough that one core group can finish inside the SLO without a batcher. Many enterprise embedding models, intent classifiers, and PII detectors live here. You gain simpler packaging, cheaper overflow capacity, and the ability to run next to the application in the same region.

CPU also fits for offline or nearline batches: nightly embedding backfills, corpus rebuilds, and evaluation sweeps that care about throughput per dollar of cores rather than tokens per second on a decoder. Pin thread counts and NUMA. A noisy neighbor on the same socket will look like “CPU is too slow” when the real issue is contention.

Quantization can help a small graph. It does not turn a large decoder into a CPU product. If you need 4-bit tricks, KV paging, and continuous batching to hit the SLO, you already left the CPU lane. Dedicated private AI infrastructure is still useful on the CPU side when embeddings sit beside regulated indexes and cannot ride a public embedding API.

When does inference still require a GPU?

GPU fits when each user token walks a large residual stream. Chat, agents, and long-context summarization spend their time in decode. CPU SIMD will not hide that. If Time to First Token and inter-token delay are in the product contract, start on exclusive GPUs and only move stages off after you measure them.

GPU also fits when batching many live decodes is the economics. A partially filled GPU can still beat a rack of cores on a 7B-and-up decoder. The opposite is also true: a half-idle flagship GPU serving a 300M encoder is a capacity bug, not a platform standard.

Regulated teams should treat GPU placement as a data-path decision too. Weights, prompts, and logs stay in the same control boundary as the card. Shared public inference APIs are a different product, even if the model name matches.

How do you prove the split without a religious argument?

Pick one production trace. Replay the embedding stage on CPU and the generator on GPU, then invert each stage only if the SLO still holds. Record p50 and p95, not a single laptop demo. Freeze batch size and context length or the comparison is fiction.

Watch utilization with the right unit. CPU jobs care about core steal and memory bandwidth. GPU decode jobs often look “underutilized” while the batcher is waiting on users. Do not move decode to CPU because SM% is 40. Do not move embeddings to GPU because a dashboard said GPUs must be busy.

OneSource Cloud runs exclusive GPU pools in U.S. facilities, including Texas / Richardson, for the stages that actually need them. OnePlus Platform, OneSource Cloud's AI orchestration platform, should schedule CPU workers and GPU replicas as different quotas so a backfill cannot evict chat. If your graph is retrieval-heavy, keep document stores on the AI storage architecture path you already trust, and do not pull that I/O onto the GPU node as a side effect.

FAQ

Can we serve a 7B or 8B chat model on CPU?

You can start a process. Hitting a multi-user SLO is the real test. If you need aggressive quantization, tiny context, and almost no concurrency, CPU may limp through a pilot. Production chat with tools usually needs a GPU. Measure p95 tokens, not a single prompt.

Are embeddings always cheaper on CPU?

Often, for moderate QPS and medium-size encoders. Very large embedding models or tight millisecond SLOs can still want a GPU. Compare cost per million vectors at the batch size you will run, on exclusive hardware, not on a laptop with turbo clocks.

Does this change if we use INT8 or GGUF?

Lower precision can move a borderline graph. It does not rewrite the physics of a large decode loop. Treat quantized CPU serving as a candidate to measure, not as a policy. Keep a GPU fallback until the SLO holds for a full week of traffic.

Should training stay on GPU even if inference moves to CPU?

Yes for any serious LLM or large encoder train. This page is inference only. Do not use a CPU inference win to justify CPU training. Fine-tunes and full-weight jobs belong on the GPU pools you already isolated.

How do we stop teams from defaulting everything to H100?

Publish a placement table and a quota. Small graphs get CPU by default. Decoders get GPU by default. Review exceptions with a trace. Managed AI infrastructure helps when someone must enforce that table after the first outage.

Summary

Put small, wide, or batch-friendly stages on CPU. Put large autoregressive decode on exclusive GPUs. Prove the split with a production trace at fixed context and batch, and do not use SM% or a quantization slogan as the only evidence.

Hybrid graphs are normal. Exclusive U.S. GPU capacity should cover the stages that need it, not every microservice that mentions a model. If you need help naming those pools, start from OneSource Cloud private AI infrastructure and keep CPU workers out of the GPU reservation.