Cascaded vs Speech-to-Speech Voice Agents for Inference

Quick Verdict: Keep a cascaded voice agent when you need inspectable text, separate vendors, and independent upgrades of ASR, LLM, and TTS. Choose speech-to-speech when turn latency and prosody matter more than a readable transcript as the system of record.

A cascaded voice agent is an inference pipeline that turns speech into text, runs a language model, then synthesizes audio, while a speech-to-speech agent is a model path that maps audio to audio without making text the only intermediate. Both still need GPUs. They occupy them differently.

This page is an architecture choice. It is not a voice-network latency budget and not a general LLM serving guide. Those pages assume you already picked the stack.

What does each path optimize?

Cascaded systems optimize control. You can log the transcript, apply a policy model on text, and swap TTS without retraining the LLM. You also stack three failure modes and three tail latencies. The user hears the sum.

Speech-to-speech optimizes the conversation as audio. Barge-in and emotion ride the same model. You lose a cheap place to put a rules engine unless you add a parallel ASR just for logs, which quietly becomes a cascade again.

How should GPU capacity follow the choice?

Cascaded pools can specialize: ASR on one SKU, LLM on another, TTS on a third. That helps utilization and hurts operational count. Speech-to-speech often wants a single hot replica that holds the full stack in memory. Idle speech models are expensive because you cannot hide them behind a text cache as easily.

Measure what the caller hears: time to first audio and time to a barged interruption. Do not declare a winner from LLM tokens per second. A fast LLM behind a slow TTS still sounds late. Exclusive GPUs on private AI infrastructure keep a voice pool off a training neighbor. They do not pick the architecture.

Which risks are easy to miss?

Cascaded stacks leak policy through the transcript store. If that store is the compliance record, treat it as production data, not a debug file. Speech-to-speech stacks leak policy through the audio itself. You still need a review path, even if it is sampled human listening rather than grep.

OneSource Cloud can host either pool in U.S. facilities, including Texas / Richardson. OnePlus Platform, OneSource Cloud’s AI orchestration platform, can keep the voice project off research quotas. Use that for isolation. Do not use it as a reason to skip a listening test.

FAQ

Can we start cascaded and add speech-to-speech later?

Yes, if you kept audio, not only transcripts. A text-only archive cannot retrain or evaluate a speech-to-speech model. Budget storage for audio from the first pilot if that path is on the roadmap.

Does speech-to-speech remove the need for an LLM?

Not always. Many products still call tools, fetch records, or draft text. The speech model may wrap an LLM or emit text on the side. Count the GPUs for every model that stays in the turn.

Which option is cheaper to serve?

It depends on overlap and idle time. Three small specialized replicas can beat one large speech replica, or the reverse, once you include the human cost of operating three services. Price the occupancy of the peak hour, not a brochure FLOP count.

Do contact centers have to stay cascaded?

No, but they often stay cascaded because supervisors need text, and because CRM tools expect strings. If you move to speech-to-speech, design the supervisor view before you design the model demo.

Summary

Cascaded voice agents buy inspectable text and swappable parts at the cost of stacked latency. Speech-to-speech agents buy a more natural turn at the cost of a heavier, less modular GPU footprint. Pick the contract you can operate, then size the pool to first audio, not to a text-only benchmark.

When the voice pool needs exclusive GPUs, review OneSource Cloud home, private AI infrastructure, and managed AI infrastructure rather than parking a speech replica on a shared training node.