How Full-Duplex Voice Agents Work: Splitting Talking from Reasoning in Real-Time AI
An architectural deep dive into low-latency full-duplex voice systems, continuous audio token streaming, endpointing, and dual-model orchestration.
On this page
Traditional voice assistants operate like walkie-talkies. You speak into a microphone, wait for an automatic speech recognition (ASR) engine to transcribe audio into text, wait for an LLM to generate an answer, and wait for a text-to-speech (TTS) engine to synthesize audio. The interaction is turn-based, stiff, and suffers from a compounding 1.5 to 3-second latency penalty. Worse, if you speak while the assistant talks, it cannot hear you; the microphone is effectively muted during playback.
Full-duplex conversational voice systems eliminate turn-taking. They process audio streams bidirectionally in 20-millisecond frames, simultaneously listening while speaking, detecting natural human interjections (“yeah”, “uh-huh”), and handling abrupt user barge-ins within 200 milliseconds. Building this requires splitting conversational flow into two distinct architectural loops: a fast, stateful audio loop running at a fixed clock, and an asynchronous reasoning and tool-execution engine.
The Three Generations of Voice AI Architecture
Voice architectures have evolved across three distinct generations, trading pipeline simplicity for biological conversational fidelity:
[Generation 1: Cascaded Pipeline (Turn-Based)]
User Audio -> [ASR] -> Text -> [LLM] -> Text -> [TTS] -> Audio Output
Latency: 1,800ms - 3,200ms | Tone Lost | No Interruptibility
[Generation 2: Native Audio-to-Audio (Turn-Based)]
User Audio -> [Multimodal Speech LLM] -> Streamed Audio Tokens
Latency: 600ms - 1,000ms | Emotive Tone Preserved | Half-Duplex (Muted when speaking)
[Generation 3: Full-Duplex Split Loop (Live Realtime)]
User Stream <=== Bidirectional 20ms Frames ===> [Real-Time Audio Worker]
^
Async Context Sync | Tool Results
v
[Frontier Reasoner]
Latency: 180ms - 250ms | Instant Barge-In | Complex External Execution
Cascaded pipelines discard emotional nuance, hesitation, and pitch because text transcription strips vocal prosody. Generation 2 models ingest audio tokens directly and generate audio tokens natively, preserving inflection, but they remain frozen in a half-duplex turn structure. Generation 3 splits conversational execution across two coordinated systems.
Splitting Talking from Reasoning: The Dual-Loop Engine
The central engineering challenge of real-time voice is that human conversation requires 200ms acoustic responsiveness, whereas enterprise reasoning and tool execution often require 1 to 3 seconds of compute time.
If a single model must do both, you either sacrifice conversational responsiveness or cripple intelligence. Full-duplex systems resolve this via Functional Decomposition:
+-----------------------------------------------------------------------------+
| FULL-DUPLEX DUAL ARCHITECTURE |
+-----------------------------------------------------------------------------+
[User Mic] === 20ms Opus Audio Frames ===> [Edge WebSocket / WebRTC Server]
|
+----------------------------+----------------------------+
| |
v (Continuous 50Hz Clock) v (Event Triggers)
+------------------------------------+ +----------------------------------+
| FAST AUDIO LOOP (Stateful Worker) | | ASYNC REASONING PATH (Frontier) |
| - Native Audio-to-Audio Model | | - Massive Reasoning Model |
| - Low latency prosody generation | | - Multi-step Tool Executions |
| - Acoustic barge-in detector | | - Enterprise Database Lookups |
| - Latency: 180ms to audio output | | - Output: Background Injections |
+------------------------------------+ +----------------------------------+
^ |
|============= Context Patch & Tool Injections ===========+
1. The Fast Audio Loop (The Conversational Engine)
The fast audio model runs on a dedicated GPU instance pinned to the client’s WebRTC or WebSocket connection. It operates at 50 Hz (one forward pass every 20ms). Its role is strictly conversational:
- It generates instantaneous filler speech (“Let me pull up that invoice right now…”) without waiting for data.
- It detects pitch shifts and backchannels without breaking the flow.
- It continuously tracks acoustic power to instantly kill audio playback the moment the user interrupts.
2. The Asynchronous Reasoning Path (The Deep Engine)
When the user requests an action requiring database retrieval or tool execution (“What did we bill ACME Corp last Tuesday?”), the fast loop emits a structured tool trigger to an asynchronous reasoning queue. The frontier model executes the query, formats the facts into a concise grounding delta, and patches the fast model’s KV cache. The fast model effortlessly shifts from filler cadence into factual delivery.
Engineering the Live Path: Audio Streaming and WebRTC
Live audio cannot be routed through stateless HTTP request-response proxies. Full-duplex requires continuous, stateful media transport:
[Client Microphone]
| (Opus Audio: 24kHz, 16-bit PCM, 20ms packets)
v
[WebRTC Media Gateway (ICE / STUN / TURN)]
| (RTP Media Stream)
v
[Inference Worker Host (Shared Memory Ring Buffer)]
| (Direct GPU Memory Copy via CUDA IPC)
v
[Speech Transformer Audio Encoder]
To maintain continuous sub-200ms latency, the inference engine eliminates socket serialization overhead:
- Zero-Copy Audio Buffers: Audio frames landing on the host NIC are transferred directly into GPU memory via CUDA Inter-Process Communication (IPC) without transiting through user-space Python runtime memory.
- Continuous Prefilling: As the user speaks, their audio tokens are prefilled into the transformer’s KV cache in real time. When the user finishes speaking, the model does not need to digest a 5-second prompt; the KV cache is already 99% populated, allowing first-token audio generation within 80 milliseconds.
The Endpointing and Barge-In Challenge
In traditional systems, silence detection (Voice Activity Detection, or VAD) decides when a user has finished speaking. If the user pauses to think for 400 milliseconds, naive VAD prematurely cuts them off and triggers an unwanted model response.
Full-duplex systems discard volume-based VAD in favor of Semantic Acoustic Endpointing:
Endpoint Probability = Model_Head(Acoustic Pitch, Syntactic Completeness, Breathing Cadence)
- Syntactic and Acoustic Cues: When a user drops their pitch at the end of a declarative sentence, the model registers a high endpoint probability and begins speaking. When the user trails off with rising inflection (“I was looking for… um…”), the model detects an incomplete thought and holds its silence, even across 800ms of dead air.
- Instant Acoustic Barge-In: The moment incoming microphone energy exceeds room ambient noise while the agent is speaking, the edge worker immediately drops downstream audio packets and halts the TTS generation queue, clearing the buffer in under 40 milliseconds. The user never feels the jarring sensation of talking over a recorded machine.
Architectural Takeaway
Full-duplex voice cannot be achieved by stitching disparate REST APIs together. Designing a conversational voice system requires:
- Dedicated, stateful WebRTC audio streaming pinned directly to GPU inference workers.
- Decoupling conversational cadence (fast 20ms audio-to-audio loop) from heavy cognitive processing (async tool-calling reasoner).
- Continuous stream prefilling into the attention KV cache during speech to eliminate turn-start latency.
- Semantic endpointing and instantaneous client-side audio cancellation for natural barge-in handling.
/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.