Realtime voice is the only modality where being right is not good enough — you also have to be right now. A text model can take four seconds and nobody blinks. A voice agent that takes four seconds to start speaking sounds broken. I spent a chunk of last year wiring up bidirectional audio, and most of the hard problems had nothing to do with the model.
The latency budget is a sum, and you don't control most of it
The number users feel is time to first audible byte after they stop talking. That is not one number, it's a chain:
mic capture + encode 15-55 ms
network uplink RTT/2 20-150 ms (mobile: worse, and variable)
VAD end-of-turn hold 200-700 ms <-- you own this one
server queue + model 150-600 ms
first TTS chunk 50-300 ms
network downlink 20-150 ms
client jitter buffer 40-120 ms <-- you own this one too
A "fast" pipeline lands between several hundred milliseconds and over a second. Human conversational turn gaps are famously short — around 200 ms in the conversation-analysis literature — which is why even good voice agents feel slightly like a satellite call.
The two terms you control are the end-of-turn hold and the jitter buffer, and both trade latency against correctness. Shrink the hold and you interrupt people mid-sentence. Shrink the jitter buffer and you get dropouts on flaky Wi-Fi. Instrument every segment before optimizing: almost every time I've been asked to "make the voice agent faster," the culprit was a fixed 500 ms silence timeout nobody had touched, not the model.
VAD is a classifier, so it has a confusion matrix
Voice activity detection sounds like a solved DSP problem until you deploy it. Energy-threshold VAD (frame RMS against an adaptive noise floor) is free and works fine in a quiet room. It falls apart with a TV, a fan, or a café. Neural VAD — Silero and WebRTC's are the two you'll encounter — is far more robust to non-speech noise, but costs a frame or two of lookahead and still can't distinguish your speech from a nearby speaker's.
The part people underestimate: VAD errors are asymmetric in cost.
False negative (you think the user is still talking): the agent sits there. Laggy, but recoverable.
False positive (you cut them off mid-thought): the agent barges in on a pause. Users hate this far more, and it poisons the transcript because you ship a truncated utterance to the model.
So the hold timer should not be a constant. A pause after "so I was thinking, um—" is grammatically incomplete; a pause after "what's the weather tomorrow" is a complete request. If your streaming ASR emits partial hypotheses, use that text as a weak end-of-turn signal and shorten the hold when the transcript looks complete:
def end_of_turn_hold(partial_text: str, pitch_falling: bool) -> int:
hold = 640
# Complete-looking utterance -> commit sooner.
if looks_syntactically_complete(partial_text):
hold -= 220
# Falling terminal pitch is a turn-yielding cue in many languages.
if pitch_falling:
hold -= 120
# Filled pauses mean "I'm not done" -> wait longer.
if partial_text.rstrip().endswith(("um", "uh", "so", "and", "like")):
hold += 300
return max(260, hold)
Nothing fancy, and worth more than a better model.
Barge-in is a state machine problem, not an audio problem
The moment your agent can be interrupted, you have concurrency. The user starts talking while TTS audio is still in flight, and three buffers now hold stale audio: the server's generation queue, the network, and the client's playback buffer. Cancelling server-side does nothing if the client already has 800 ms queued — the agent keeps talking for almost a second after being interrupted, which reads as rude.
Cancellation has to be client-first, server-second:
on user_speech_detected(confidence):
if state == SPEAKING and confidence > BARGE_IN_THRESHOLD:
playback.flush() # local, ~0 ms, do this FIRST
transport.send({type: "cancel", turn_id: current_turn})
state = LISTENING
# Truncate history to what the user ACTUALLY heard.
history.append(turn.truncate_to(playback.played_ms))
That last line is the one everyone forgets. If you log the full generated response but the user only heard the first eight words, your history is now a lie, and the model will confidently reference things it "said" that nobody heard. Track played-milliseconds and truncate to match.
Barge-in also needs a higher confidence threshold than normal turn detection, because the biggest source of false triggers is your own output. Without echo cancellation the agent hears itself through the speaker and interrupts itself in a loop. On the web, getUserMedia with echoCancellation: true covers the common case; on speakerphone in a hard-surfaced room browser AEC struggles, and you want half-duplex gating as a fallback.
Transport: prefer WebRTC once you leave the lab
Sending PCM over a WebSocket is easy and works right up until packet loss. TCP head-of-line blocking means one lost segment stalls everything queued behind it, and your tuned jitter buffer eats a 300 ms spike. WebRTC gives you an SRTP/UDP path with real jitter buffering, loss concealment, and AEC free from the browser stack, at the cost of signaling complexity. My heuristic: WebSocket + Opus is fine for prototypes; anything shipping to mobile networks belongs on WebRTC. Either way send Opus at 16 kHz mono, not raw PCM at 48 kHz.
When I wanted a reference for how these pieces behave alongside other modalities in one session, the GeminiOmni studio was a useful system to poke at, because it surfaces the ugly interaction fast: the moment a voice turn triggers a slow non-voice tool call, your latency budget is blown and you need a filler-speech strategy or users assume it crashed.
One page for my past self: log per-segment timestamps from day one, make the hold adaptive before touching anything else, truncate history to what was played, and test on a phone, on cellular, in a room with a fan running. These systems are all beautiful on a wired desktop with a headset, and that is not where your users are.
SOCIAL SHARE CARD GENERATOR