Generative Voice Agents Move to Production: Latency, Turn-Taking, and Telephony
How generative voice agents reach production: latency budgets, turn-taking, and PSTN/SIP telephony. Architecture patterns and procurement questions for 2026.
Voice agents have crossed a threshold. Through 2024 and 2025, generative voice demos impressed conference audiences but buckled under the weight of real callers, real networks, and real compliance requirements. In 2026, that gap is closing—contact centers, healthcare schedulers, field service dispatchers, and financial services teams are running generative voice agents on live phone lines, at volume, with service-level targets attached.
The shift changes what matters. Model quality is now table stakes. The variables that decide whether a deployment survives contact with production are engineering constraints: how tightly you can control end-to-end latency, how gracefully the agent handles the messy choreography of human turn-taking, and how well the stack absorbs the realities of PSTN and SIP telephony. This article breaks down all three, plus the architecture patterns and procurement questions that separate pilots from production systems.
Why 2026 Is the Production Inflection Point for Voice Agents
From Scripted IVR to Generative Conversations
Deterministic IVR trees—press one for billing, two for support—were predictable, auditable, and universally disliked. The first wave of "conversational" IVR bolted intent classification onto those trees, which improved routing but not conversation. Generative voice agents replace the tree entirely: the caller states what they need in natural language, the agent reasons over tools and knowledge sources, and the dialogue path is constructed in real time.
That flexibility is exactly what makes production harder. A scripted flow fails in known ways. A generative agent fails in open-ended ways, which is why the engineering discipline around it—guardrails, evaluation, fallback paths—has matured alongside the models. (See our guide to voice agent evaluation and guardrails.)
What Buyers Now Expect
Buyers who have evaluated voice agents in the past year have converged on a baseline set of expectations:
- Sub-second perceived responsiveness to caller speech—measured as the felt pause before the agent audibly engages, not raw API latency (see the latency section for why this distinction matters)
- Tolerance for interruption—callers talk over agents, and the agent must yield cleanly
- Natural speech synthesis without the robotic cadence of legacy TTS
- Graceful escalation to a human with full context when the agent is out of its depth
None of these are model problems. They are systems problems, and they are where deployments succeed or stall.
Latency: The Budget That Decides Voice Agent Viability
Decomposing the End-to-End Latency Budget
Every voice turn is a pipeline, and every stage spends latency. The budget below reflects typical production ranges; treat the numbers as planning anchors to benchmark against your own stack, not vendor guarantees:
| Stage | Typical contribution (P50 → tail) |
|---|---|
| VAD / endpointing detection window | 150–500 ms (deliberate silence before the agent decides the caller is done) |
| ASR partial/final transcript | 100–300 ms |
| LLM time-to-first-token | 200–600 ms (higher on long-context or tool-calling turns) |
| Speech synthesis time-to-first-byte | 100–300 ms |
| Network and telephony transport (round trip) | 50–200 ms |
| Audio jitter/playout buffer | 20–100 ms |
Sum these serially and a caller hears roughly a second or more of silence before the agent responds—enough to trigger "hello? are you there?" and a cascade of turn-taking failures. The engineering work is in two directions: overlapping stages so synthesis begins on the first LLM tokens rather than waiting for a complete response, and trimming each stage through model selection, regional deployment, and connection reuse. Note that the endpointing window is a deliberate cost—shortening it trades latency against the risk of cutting callers off mid-thought (see turn-taking). (Related: how to build a low-latency voice AI stack.)
Streaming Speech Synthesis and Realtime APIs
The single biggest latency win is replacing request-response chains with streaming pipelines. Older architectures waited for a complete transcript, then a complete LLM response, then a complete audio file. Modern stacks stream at every hop: ASR emits partial hypotheses, the LLM streams tokens, and speech synthesis begins generating audio from the first clause while the rest of the response is still being generated.
Realtime speech-to-speech APIs push this further by collapsing ASR, reasoning, and synthesis into a single low-latency session, removing serialization overhead between stages. The tradeoff is control: fully integrated realtime models can be harder to instrument, harder to constrain with tool calls, and harder to swap—and, critically for regulated buyers, harder to fit into audit logging, PII redaction, and data-residency requirements, since the audio-to-audio path can obscure where data is processed and retained. Many production teams run a hybrid: realtime APIs for the conversational fast path, with orchestrated tool calls and retrieval handled out-of-band so they don't block audio, and with logging and redaction applied at the orchestration layer rather than inside the realtime session.
Measuring What Matters: P50 vs. P95 vs. Perceived Responsiveness
Averages lie. A P50 latency of 700 ms sounds acceptable until you see the P95 sitting at 2.4 seconds—and callers don't experience averages, they experience the worst turns. Tail latency drives abandonment, frustration, and premature hang-ups, so production SLAs should be written against P95 and P99, not means. (The figures above are illustrative; the discipline of measuring tails is what matters, not the specific numbers.)
Perceived responsiveness is a separate metric from measured latency. Filler speech ("let me pull that up"), early acknowledgment tokens, and starting audio before the full answer is composed all reduce the felt pause even when the underlying computation takes the same time. Instrument both: raw stage-level latency for engineering, and caller-perceived responsiveness—via interruption rates, silence-triggered reprompts, and abandonment—for product. (Dive deeper: voice agent latency benchmarks and SLAs.)
"The agent that answers in 900 milliseconds but never leaves dead air outperforms the agent that answers in 600 milliseconds and stalls on every third turn."
Turn-Taking: The Hardest Unsolved Problem in Voice AI
Endpointing, Barge-In, and Backchanneling
Turn-taking is where voice agents most visibly fail, because human conversation runs on cues that are easy to produce and hard to detect.
- Endpointing is the decision that the caller has finished speaking. Too eager, and the agent interrupts a caller who merely paused to think; too patient, and the agent leaves dead air. Simple voice-activity detection (VAD) keys on silence duration alone and breaks on callers who pause mid-sentence. Better systems add semantic endpointing—a lightweight model that weighs prosody, partial transcripts, and syntactic completeness to predict whether the utterance is actually done—at the cost of a few hundred milliseconds of additional deliberation.
- Barge-in is the caller talking over the agent. Clean barge-in requires acoustic echo cancellation so the agent doesn't hear its own voice, low-latency VAD to detect the interruption, and the ability to stop synthesis immediately mid-stream rather than finishing the sentence. Agents that can't truncate audio talk over callers and feel dominant and broken.
- Backchanneling is the "mm-hmm," "right," and "okay" that signal listening without seizing the turn. It is the least mature capability and the one most often skipped; without it, agents feel like they're waiting their turn in a script rather than listening.
The hard part is that these three interact. Aggressive barge-in detection raises false triggers (the agent stops for a cough); conservative endpointing raises dead air. Production teams tune these jointly and measure both interruption false-positives and silence-triggered reprompts as a single trade-off curve.
Telephony: Where the Stack Meets the PSTN
Generative models don't answer phones—telephony infrastructure does. The last mile is where latency, audio quality, and compliance are quietly won or lost.
- Codecs and transcoding. The PSTN and most SIP carriers deliver narrowband audio (G.711 at 8 kHz). If your ASR and TTS models expect wideband or 16 kHz input, every call pays a resampling/transcoding cost and loses fidelity that degrades transcription accuracy. Match codecs end-to-end where possible and budget the transcoding hop explicitly.
- Jitter buffering and packet loss. Real networks jitter and drop packets. The playout buffer that smooths this adds latency you didn't plan for; on poor mobile connections it can swing by hundreds of milliseconds. Design for the bad network, not the lab.
- Media anchoring and regions. Where the carrier anchors media (and whether it hairpins through a distant region) can dominate your transport latency. Co-locate your agent infrastructure near the carrier's media edge, and confirm the carrier's region topology during procurement rather than discovering it in the P95.
- Signaling and call control. SIP INVITE handling, call transfer, DTMF collection, and warm handoff to a human all have their own failure modes. A generative agent that can't cleanly execute a warm transfer with context intact is not production-ready regardless of its latency.
- Compliance surface. Call recording consent, PII handling in transcripts, retention, and data residency are telephony-layer obligations as much as model-layer ones. Confirm where audio is recorded, where it is stored, and how redaction applies before the first live call.
Architecture Patterns: From Pilot to Production
Three patterns dominate production deployments. Most teams land on a hybrid.
- Orchestrated pipeline. Separate ASR, LLM, and TTS services stitched together by an orchestrator. Maximum control, observability, and swappability; highest integration and latency-tuning burden.
- Realtime speech-to-speech. A single low-latency session handling audio in and out. Lowest latency and simplest plumbing; least control, weakest instrumentation, and the hardest to fit into strict compliance regimes.
- Hybrid. Realtime session for the conversational fast path, with tool calls, retrieval, logging, and redaction handled out-of-band. Balances responsiveness against control and auditability—the pattern most regulated buyers converge on.
Procurement Questions That Separate Pilots From Production
Ask vendors these before signing, and expect specific answers:
- How do you measure and report latency—P50, P95, or means—and at which stage boundaries? A vendor quoting only P50 is hiding the number that matters.
- Can the agent truncate synthesis mid-stream on barge-in, and how do you handle false triggers?
- Where is audio processed and stored, and how does that map to our data-residency and retention obligations?
- What is the warm-transfer path to a human, and what context travels with the call?
- Which codecs and carrier regions do you support, and where does media anchor relative to your compute?
- How do you evaluate turn-taking quality in production—what do you instrument, and what does "good" look like on your dashboards?
The teams that get generative voice agents into production in 2026 won't be the ones with the best model. They'll be the ones that treated latency, turn-taking, and telephony as first-class engineering problems—and held their vendors to the same standard.