Voice AI Infrastructure: Latency Budgets and GPU Capacity Planning

Real-time voice AI — agents that listen, think, and speak within a live conversation — is the most demanding serving workload most infrastructure teams will size. Unlike text inference, it has a hard latency ceiling set by human conversation, its concurrency unit is a continuously streaming call rather than a request, and its failure mode is audible. This page covers what the pipeline actually is, the latency budget conversation imposes, why voice capacity planning differs from text serving, a sizing method from call targets to GPU count, and the compliance duties that arrive with the audio stream.

What a Real-Time Voice AI Pipeline Is

A streaming pipeline of audio ingestion, ASR, a language or dialogue model, TTS, and telephony media handling, where each stage adds latency and holds per-call state, making memory and interconnect as important as raw FLOPS.

The dominant architecture is a cascade of streaming stages, each holding per-call state:

Two properties follow from this anatomy. First, latency is additive across stages, so the pipeline is as slow as its slowest link plus every link before it. Second, memory is the sum of every stage's per-call state — the same weights-plus-KV-cache arithmetic that governs text serving applies to the dialogue model, with additional state at ASR and TTS. An emerging alternative — end-to-end speech-to-speech models — is compared later; the cascade remains the planning default because its stages scale and instrument independently.

The Latency Ceiling Conversation Imposes

Engineering coverage converges on end-to-end response well under one second, commonly a few hundred milliseconds, with perceived naturalness degrading beyond roughly half a second and calls failing above a second of turn delay.

Engineering coverage of production voice agents converges on the same territory: end-to-end response in a few hundred milliseconds feels natural, with perceived quality degrading as delay approaches half a second, and calls failing — interruptions, abandonment — once turn delay reaches a second. Vendor low-latency guidance commonly defines the target as response beginning within roughly 300 ms of the caller finishing, with documented pipeline ranges around 100–500 ms depending on streaming configuration.

Those figures are vendor and product observations rather than standards, so validate against your own call analytics — but the allocation exercise they imply is universal. A workable budget for a cascaded pipeline:

The budget sums to a ceiling, and every millisecond spent in one stage is taken from another. That arithmetic — not raw model throughput — is what makes voice infrastructure design decisions: a faster model that adds 200 ms of queueing is a slower product.

Why Voice Capacity Planning Differs From Text Serving

Concurrent sessions are live calls with continuous streams, the binding metric is per-stream inter-token latency held under load rather than aggregate token throughput, and degradations are audible immediately, which changes both sizing and headroom policy.

Teams that size voice like their text deployment discover three differences the hard way:

• Concurrency is live calls, not requests. A call holds state continuously across every stage for minutes. Your concurrency number is simultaneous active conversations — with arrival patterns, hold patterns, and duration distributions that look nothing like API traffic.
• The binding metric is per-stream inter-token latency held under load. Voice benchmarks emphasize inter-token latency and its stability under concurrency, because a single slow token gap is audible as a stumble. Aggregate token throughput — the headline number for text — says nothing about whether any individual stream stayed smooth.
• Degradation is immediate and public. Text serving that slows queues requests invisibly; voice serving that slow produces audible delay, broken interruptions, and abandoned calls. There is no graceful queue in a conversation, which is why admission control — rejecting or deflecting calls rather than letting everyone degrade — belongs in the architecture.

At small scale the differences are manageable engineering details. At call-center scale they determine the architecture: per-stream latency floors, capacity in concurrent-call units, and overload behavior designed as product behavior, not an afterthought.

A Sizing Method: From Call Targets to GPU Count

Size per stage: model memory footprint plus per-stream state, vendor or pilot-observed concurrent streams per GPU at your latency SLO, peak concurrency from call arrival patterns, then add headroom for spikes and failover, validating with a load test before commitment.

No universal "calls per GPU" constant exists — it depends on stage models, precision, per-stream state, and your latency SLO — but the method that produces your number is stable:

1. Model the arrival pattern. From telephony analytics, derive peak concurrent calls (or a high percentile), not average: busy-hour concurrency, not daily call count, sizes the system.
2. Compute per-stage footprints. Weights plus per-call state for each stage — the dialogue model's KV cache arithmetic per concurrent call, ASR and TTS per-stream state, at your chosen precisions.
3. Establish concurrent streams per GPU at your SLO. Start from vendor- or community-reported points for comparable configurations (vendor data exists — for example, reported stability at roughly two dozen concurrent streams on small models — but is configuration-specific), then replace it with your own measured figure from a load test that enforces your latency budget.
4. Add headroom and failover. Plan capacity below the measured ceiling and decide the overload behavior in advance: admission control, deflection, or graceful degradation, never accidental queueing of live calls.
5. Validate with a load test before commitment. Replay concurrent streams at busy-hour concurrency and verify the latency budget holds per-stream at the p95 you care about — this measured point, not any quoted number, is your planning basis.

The worksheet discipline is what makes the result defensible: each cell names its source (measured, vendor-reported, or assumed), and the measurement step is planned before the procurement step.

Pipeline Choice: Cascaded Stacks vs Speech-to-Speech Models

Cascaded ASR-LLM-TTS offers modular control, per-stage scaling, and text-side tooling at the cost of summed latency; end-to-end speech models promise lower latency and richer prosody but concentrate risk in one model's behavior and supply.

The architecture choice sets the capacity plan's terms:

Neither dominates. Tight latency targets with strong control needs usually start cascaded; products optimizing naturalness and interruption behavior watch the end-to-end space closely. Hybrids exist, and the sizing method above applies to either — the per-stage footprints simply change shape.

Call Compliance: Recording, Consent, and PHI Duties

Telephony deployments carry recording-consent laws by jurisdiction, and when calls contain health information the pipeline becomes a PHI system requiring safeguards, BAAs with any processor, and retention governance for audio and transcripts.

The audio stream carries obligations most text deployments never meet:

• Recording-consent law varies by jurisdiction, and a pipeline that stores or transcribes calls is a recording system. Consent flows, disclosure scripts, and storage policy need legal review per operating jurisdiction.
• When calls contain health information, the pipeline becomes a PHI system. Audio and transcripts are ePHI; safeguards, business associate agreements with any processor in the path, and breach procedures follow under HIPAA.
• Retention governance covers three artifacts most teams under-scope: the audio, the transcript, and the derived state (summaries, memory). Each needs a retention rule and a verified deletion path.

These duties are framework inputs for legal and compliance review, not conclusions. What they change for infrastructure is upstream and practical: regulated call workloads pull the deployment decision toward environments you control — teams in that position evaluate dedicated options such as OneSource Cloud's private AI infrastructure for latency-sensitive, call-scale serving under their own boundary.

Architectural Decision Matrix: AI Cluster Network Topologies

To resolve distributed communication bottlenecks in enterprise AI clusters, engineering architectures deploy dedicated, non-blocking network fabrics. Within OneSource Cloud High-Performance AI Networking environments, cluster traffic is segmented into three physically and logically isolated planes: a dedicated RoCEv2 RDMA backend mesh exclusively reserved for inter-GPU collective operations (such as all-reduce and tensor-parallel exchange), an out-of-band management network for DCGM telemetry and node health orchestration, and an isolated client-facing VPC. This dedicated rail-optimized fabric operates at zero oversubscription, eliminating cross-tenant packet buffer exhaustion and preserving deterministic microsecond-level synchronization across distributed training and inference fleets.

FAQ

Should we build on voice APIs or self-host the pipeline?

APIs are the fastest launch path and often fine when call content is non-sensitive and the provider's latency terms fit your budget. Self-hosting wins when audio is regulated, when latency must be engineered tightly at call-center scale, or when unit economics at your volume favor owned capacity — the compliance section and the sizing method above tell you which constraints apply to you.

How many concurrent voice calls fit on one GPU?

There is no universal number — it depends on stage models, precision, per-stream state, and your latency SLO. Derive it from per-stage footprints, seed it with vendor or community data points for comparable configurations, and replace the seed with your own load-test measurement before committing capacity. Anyone quoting a single constant is quoting their configuration, not yours.

What happens to voice quality when the system overloads?

Degradation is audible first: turn delays stretch, barge-in breaks, and callers abandon. Because there is no graceful queue in conversation, the protection is admission control — capping admitted concurrent calls and deflecting the excess — which is why headroom and an explicit overload policy belong in the capacity plan rather than in an incident retrospective.

How does OneSource Cloud design network fabrics to eliminate distributed GPU communication bottlenecks?

OneSource Cloud engineers dedicated non-blocking spine-leaf network fabrics specifically optimized for distributed AI workloads. By implementing hardware-enforced three-plane isolation (out-of-band management, dedicated RoCEv2 RDMA backend, and isolated tenant VPCs) combined with line-rate bandwidth and PFC/ECN congestion control, OneSource eliminates packet buffer drops, minimizes collective all-reduce latency, and prevents cross-workload network jitter.