How OpenAI Scales Real-Time Voice WebRTC to 900 Million Users
How OpenAI solved Kubernetes port exhaustion and state stickiness by decoupling WebRTC into stateless UDP relays and stateful GPU media transceivers.
On this page
When OpenAI rolled out Advanced Voice Mode to hundreds of millions of weekly active users, the primary engineering crisis was not model training. The crisis was networking infrastructure.
Voice AI requires ultra-low latency: audio packets must travel between the user’s phone and cloud GPUs in under 100 milliseconds. Standard HTTP/WebSocket streaming is inadequate because TCP’s guaranteed delivery mechanics cause head-of-line blocking: if one audio packet drops on a cellular network, all subsequent packets stall while TCP retransmits.
To achieve real-time streaming, OpenAI turned to WebRTC (Web Real-Time Communication) running over UDP. But operating WebRTC at the scale of 900 million users broke standard Kubernetes container networking assumptions.
To survive, OpenAI decoupled the stateful WebRTC protocol into a Stateless UDP Relay Layer and an Independent Stateful Transceiver Layer.
The Networking Dilemma: TCP vs WebRTC (UDP)
+-----------------------------------------------------------------------+
| WebSocket over TCP: Brittle on Cellular Connections |
| - Head-of-Line Blocking: A single dropped packet stalls the stream |
| - Audio buffers stutter; latency spikes from 100ms to 2,000ms |
+-----------------------------------------------------------------------+
| WebRTC over UDP: Built for Real-Time Human Media |
| - SRTP Encrypted Audio Frames; Drops lost packets rather than stall |
| - Built-in Congestion Control (BWE) & Adaptive Jitter Buffers |
| - Interactive P2P / Client-Server Latency (< 100ms) |
+-----------------------------------------------------------------------+
While WebRTC is the gold standard for video conferencing (Zoom, Google Meet), running it inside containerized cloud clusters introduces two severe infrastructure bottlenecks:
- UDP Port Exhaustion: A standard WebRTC peer connection allocates dedicated UDP ports for RTP media. A single Linux host running thousands of pods quickly exhausts available ephemeral UDP ports (ports 1024 to 65535).
- Stateful Connection Stickiness: WebRTC handshakes (ICE candidate gathering and DTLS cryptographic session establishment) bind a client’s IP and port to a specific memory process. If Kubernetes load balancers route packets to a different container replica, the session immediately terminates.
Architecture: Splitting Stateless Relay from Stateful Transceiver
OpenAI resolved this by completely separating network packet reception from media processing:
+-----------------------------------------------------------------------+
| OpenAI WebRTC Edge Architecture |
+-----------------------------------------------------------------------+
| |
| [ User Mobile App / Browser (WebRTC Client) ] |
| | (Audio over SRTP / UDP) |
| v |
| [ Anycast Edge BGP Border Routers ] |
| | (Routes to nearest regional datacenter PoP) |
| v |
| [ Stateless UDP Relay Pool (Go / eBPF / XDP) ] |
| | - Binds to fixed, multiplexed UDP VIP ports |
| | - Extracts session ID from SRTP header in < 10 microseconds |
| | - Zero cryptographic or audio processing at edge |
| v |
| [ Internal High-Speed Overlay Network (InfiniBand / NVLink) ] |
| | (Routes deterministically based on consistent session hash) |
| v |
| [ Stateful Media Transceiver Pods ] |
| | - Manages DTLS session keys and SRTP decryption |
| | - Opencodec decompression -> Raw PCM audio tokens |
| v |
| [ GPU Worker Fleet: Omnidirectional Voice Model (Audio-to-Audio) ] |
| | |
| v (Synthesized audio frames streamed back across same path) |
| [ Low-Latency Audio Returned to User in < 280ms ] |
+-----------------------------------------------------------------------+
1. The Stateless UDP Relay Layer
Instead of having Kubernetes pods bind random UDP ports directly, OpenAI deploys a pool of lightweight, stateless UDP relays at the network perimeter:
- The relays listen on fixed, well-known UDP ports.
- Using eBPF and XDP (eXpress Data Path) kernel hooks, the relay inspects the unencrypted header of incoming UDP packets to read the session identifier.
- The relay forwards the raw packet directly across internal VPC networks to the designated backend pod without decrypting media or maintaining session state.
If a relay node crashes, any other relay node instantly absorbs the traffic without terminating active user sessions.
2. The Stateful Transceiver Layer
The heavy computational work: DTLS handshake negotiation, SRTP cryptographic decryption, adaptive jitter buffer management, and audio decoding (Opus codec to raw PCM): is isolated in dedicated Transceiver pods.
These pods run in close network proximity to the GPU inference fleet. When raw PCM audio frames are decrypted, they are streamed directly into GPU High-Bandwidth Memory via memory-mapped IPC buffers.
3. Graceful Cellular Migration (ICE Restart)
When a user on a mobile phone walks out of their house, transitioning from Wi-Fi to 5G cellular, their IP address and NAT port change instantly.
OpenAI’s WebRTC implementation leverages ICE Restarts:
- The client detects the network interface switch and emits an updated ICE candidate.
- The stateless relay associates the new IP/port mapping with the existing session ID.
- The audio stream continues seamlessly without a dropped syllable.
Key Engineering Takeaways
- TCP is the wrong protocol for real-time audio: Never use WebSockets for interactive voice agents. Packet drops cause head-of-line blocking that ruins conversational timing.
- Decouple stateless relays from stateful transceivers: Do not force container pods to manage ephemeral UDP ports directly. Use multiplexed edge relays to bypass Kubernetes port exhaustion.
- Handle cellular network handoffs with ICE restarts: Mobile users constantly transition between Wi-Fi and 5G. Robust WebRTC implementations must support dynamic connection re-anchoring.
/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.