Voice Agent Turn-Taking: Stop Live AI Calls From Talking Over Users

Voice agents do not usually fail because the model is “not smart enough.” They fail in the awkward half-second where the user pauses, breathes, corrects themselves, or interrupts while the AI is still talking.

That tiny moment decides whether the product feels useful or robotic.

If your live AI call cuts people off, talks over them, ignores barge-in, or waits so long that users repeat themselves, no prompt will save the experience. The fix is not one magic model. It is a turn-taking system: audio signals, semantic checks, interruption rules, streaming, and metrics that work together.

This guide walks through a practical voice agent turn-taking design you can ship in a real product.

Why turn-taking is the real voice agent bottleneck

Text chat is forgiving. A user types. The model answers. If the response takes two seconds, the user may still wait.

Voice is different. Humans expect conversation to move quickly. A delay feels like confusion. An early response feels rude. Talking over the user feels broken.

A production voice agent has to answer three questions again and again:

1. Is the user still speaking?


2. Is the user finished enough for the agent to respond?


3. If the user interrupts, should the agent stop, listen, or continue?

Most teams start with a simple pipeline:

Microphone -> Speech-to-text -> LLM -> Text-to-speech -> Speaker

That is enough for a demo. It is not enough for a live workflow where the caller changes their mind, uses filler words, speaks in a noisy room, or interrupts because the AI misunderstood them.

The practical goal is not “lowest latency at any cost.” The goal is comfortable turn timing: fast enough to feel alive, patient enough to avoid cutting people off, and interruptible enough to recover when the user takes control.

Research signals behind this topic

Recent AI platform activity points toward live agents moving from demos into production workflows:

• Product launches are emphasizing embedded live agents that can see, speak, and operate inside software.


• Voice agent platforms are highlighting same-day deployment, multilingual calls, and modular conversation blocks.


• Developer discussions keep returning to latency, context loss, governance, evaluation, and whether agents can be trusted without constant babysitting.


• Search results for voice agent latency are getting crowded, but practical implementation content around turn-taking, barge-in tuning, and interruption policy is thinner and often scattered across vendor docs.

That creates a useful content gap: builders do not only need “make it faster.” They need a concrete way to decide when the AI should speak, wait, pause, resume, or hand off.

The turn-taking stack

Think of turn-taking as a small control plane beside your voice pipeline.

Audio stream

  -> Voice activity detection

  -> Partial transcript stream

  -> End-of-turn detector

  -> Interruption policy

  -> Agent state machine

  -> Response planner

  -> Streaming TTS

Each layer answers a different question.

You can buy pieces of this stack from providers, but you still need product-specific policy. A banking workflow, medical triage flow, coding assistant, and onboarding agent should not share the same interruption behavior.

A simple state machine for live calls

Start with explicit states. Do not let every async callback mutate the session freely.

LISTENING

  user speech starts -> USER_SPEAKING



USER_SPEAKING

  possible end detected -> THINKING

  noise rejected -> LISTENING



THINKING

  response ready -> AGENT_SPEAKING

  user resumes -> USER_SPEAKING



AGENT_SPEAKING

  user barge-in -> INTERRUPTED

  speech complete -> LISTENING



INTERRUPTED

  stop TTS -> USER_SPEAKING

A basic TypeScript sketch:

type CallState =

  | 'LISTENING'

  | 'USER_SPEAKING'

  | 'THINKING'

  | 'AGENT_SPEAKING'

  | 'INTERRUPTED';



type Event =

  | { type: 'speech_started'; at: number; confidence: number }

  | { type: 'speech_ended'; at: number; transcript: string }

  | { type: 'partial_transcript'; text: string; at: number }

  | { type: 'barge_in'; at: number; confidence: number }

  | { type: 'response_ready'; responseId: string }

  | { type: 'tts_done'; responseId: string };



function transition(state: CallState, event: Event): CallState {

  if (state === 'LISTENING' && event.type === 'speech_started') {

    return event.confidence > 0.65 ? 'USER_SPEAKING' : 'LISTENING';

  }



  if (state === 'USER_SPEAKING' && event.type === 'speech_ended') {

    return 'THINKING';

  }



  if (state === 'THINKING' && event.type === 'speech_started') {

    return 'USER_SPEAKING';

  }



  if (state === 'THINKING' && event.type === 'response_ready') {

    return 'AGENT_SPEAKING';

  }



  if (state === 'AGENT_SPEAKING' && event.type === 'barge_in') {

    return event.confidence > 0.75 ? 'INTERRUPTED' : 'AGENT_SPEAKING';

  }



  if (state === 'AGENT_SPEAKING' && event.type === 'tts_done') {

    return 'LISTENING';

  }



  if (state === 'INTERRUPTED') {

    return 'USER_SPEAKING';

  }



  return state;

}

The exact states can change, but the discipline matters. Voice systems are streaming systems. Without a state machine, you will eventually play stale audio after the user has already corrected the agent.

Use both acoustic and semantic end-of-turn detection

Silence alone is a weak signal.

A user might pause because they are thinking. They might say “I need to book a flight from Delhi to…” and pause before giving the city. If your agent jumps in, the call feels clumsy.

A better end-of-turn detector combines:

• 
  Voice activity detection: did speech stop?


• 
  Silence duration: how long has the user been quiet?


• 
  Partial transcript: does the text look complete?


• 
  Intent confidence: do we have enough slots to act?


• 
  Conversation context: did the agent ask a yes/no question or an open-ended question?

Example policy:

type TurnSignal = {

  silenceMs: number;

  transcript: string;

  intentConfidence: number;

  requiredSlotsFilled: boolean;

  lastAgentQuestionType: 'yes_no' | 'slot_fill' | 'open';

};



function shouldEndTurn(signal: TurnSignal): boolean {

  const text = signal.transcript.trim().toLowerCase();



  if (signal.lastAgentQuestionType === 'yes_no') {

    return signal.silenceMs > 250 && /^(yes|no|yeah|nope|correct|right)\b/.test(text);

  }



  if (signal.lastAgentQuestionType === 'slot_fill') {

    return signal.silenceMs > 450 && signal.requiredSlotsFilled;

  }



  const looksIncomplete = /\b(and|or|from|to|because|with|for)$/i.test(text);

  if (looksIncomplete) return false;



  return signal.silenceMs > 700 && signal.intentConfidence > 0.7;

}

This is intentionally simple. You can replace the regex with a classifier later. The key point is that end-of-turn is not only an audio problem. It is a conversation problem.

Design barge-in as a policy, not a toggle

Barge-in means the user can interrupt while the AI is speaking.

Many teams treat it as a boolean setting: on or off. That is too crude.

A production system should decide what kind of interruption happened:

1. 
   Correction: “No, I meant the other account.”


2. 
   Cancellation: “Stop.”


3. 
   Clarification: “Wait, what does that mean?”


4. 
   Background noise: another person talks nearby.


5. 
   Backchannel: “mm-hmm,” “okay,” “yeah.”

These should not all behave the same.

type BargeInDecision = 'ignore' | 'duck_audio' | 'pause_and_listen' | 'stop_and_reset';



function classifyBargeIn(text: string, confidence: number): BargeInDecision {

  const clean = text.trim().toLowerCase();



  if (confidence < 0.6) return 'ignore';

  if (/^(mm|mhm|uh huh|yeah|okay)$/.test(clean)) return 'duck_audio';

  if (/\b(stop|cancel|never mind|hold on)\b/.test(clean)) return 'stop_and_reset';

  if (/\b(no|actually|wait|i mean|that's wrong)\b/.test(clean)) return 'pause_and_listen';



  return 'pause_and_listen';

}

For long responses, consider audio ducking before a full stop. Ducking lowers the AI voice while the system decides whether the user is truly taking the turn. This avoids abrupt cutoffs when the user only says “yeah.”

Split response planning from response speaking

One common bug: the LLM generates a long answer, TTS starts streaming it, then the user interrupts, but the old response keeps leaking into the call.

Avoid this by separating the response plan from the audio stream.

Each response should have an ID, a cancel token, and a current validity check.

class SpeechController {

  private activeResponseId: string | null = null;



  start(responseId: string) {

    this.activeResponseId = responseId;

  }



  shouldPlay(responseId: string) {

    return this.activeResponseId === responseId;

  }



  cancel(responseId: string) {

    if (this.activeResponseId === responseId) {

      this.activeResponseId = null;

    }

  }

}

Before each TTS chunk plays, check whether the response is still active. If the user interrupted, drop queued chunks immediately.

This also helps when tool calls finish late. A booking lookup from the previous user intent should not speak after the user has already changed the destination.

Set latency budgets by stage

You cannot tune turn-taking with one total latency number. Break it down.

A practical first budget for a responsive voice workflow:

The trick is overlap. While the user is speaking, stream partial transcripts. While the end-of-turn detector waits, prepare likely intents. While the LLM streams, send short TTS chunks.

But be careful: overlapping work creates stale work. Every async stage needs cancellation.

Add interruption-safe tool calls

Voice agents often call tools: search a knowledge base, update a CRM, schedule a meeting, refund an invoice, or open a support ticket.

Turn-taking becomes riskier when speech triggers actions.

Use three rules:

1. No irreversible tool call from partial speech.


2. Every tool call gets a user-intent version.


3. 
   If the user interrupts before commit, pause the action.
   

type ToolRequest = {

  intentVersion: number;

  toolName: string;

  args: Record<string, unknown>;

  reversible: boolean;

};



function canExecuteTool(currentIntentVersion: number, req: ToolRequest) {

  if (req.intentVersion !== currentIntentVersion) return false;

  if (!req.reversible) return false; // require confirmation elsewhere

  return true;

}

This is especially important for builders shipping AI workflows into customer-facing products. Users speak casually. They revise themselves. Your system has to treat speech as evolving input until the turn is stable.

Measure the moments users actually feel

Average latency is not enough.

Track these metrics per conversation, per environment, and per user segment:

• 
  Time to first agent audio: from detected end-of-turn to first audible response.


• 
  False cutoff rate: user resumes within 500 ms after agent starts speaking.


• 
  Barge-in success rate: user interrupts and AI stops within target time.


• 
  Ignored interruption rate: user speaks over AI but the system continues.


• 
  Dead-air p95: long silence before response.


• 
  Repeat rate: user repeats the same intent after a bad turn.


• 
  Correction rate: user says “no,” “actually,” or “I meant.”


• 
  Tool-after-interruption incidents: tool results spoken after intent changed.

A simple event log helps:

{

  "call_id": "call_123",

  "turn_id": "turn_009",

  "events": [

    { "type": "speech_started", "t": 1200 },

    { "type": "speech_ended", "t": 4100 },

    { "type": "agent_audio_started", "t": 4620 },

    { "type": "barge_in", "t": 5300 },

    { "type": "agent_audio_stopped", "t": 5410 }

  ],

  "metrics": {

    "end_to_audio_ms": 520,

    "barge_in_stop_ms": 110

  }

}

Review bad calls weekly. The fastest way to improve turn-taking is to listen to the exact moments where the user had to repeat, correct, or wait.

Tune by conversation type

Not every turn deserves the same silence threshold.

Use different settings for different moments:

A voice agent that handles an angry support call should not interrupt like a fast command palette. Context matters.

Common mistakes to avoid

Mistake 1: Optimizing only for speed

A faster agent that interrupts users is worse than a slightly slower agent that listens well. Optimize for completed turns, not benchmark bragging rights.

Mistake 2: Using one silence threshold everywhere

A 300 ms pause may be enough after “yes.” It is not enough after “my account number is…” Use adaptive thresholds.

Mistake 3: Letting TTS queues keep playing

When the user interrupts, old audio must stop. Cancel queued chunks, tool summaries, and delayed follow-ups tied to the previous intent.

Mistake 4: Treating backchannels as full interruptions

People say “yeah,” “right,” and “mm-hmm” while listening. Do not reset the whole conversation every time.

A practical implementation checklist

Use this before you ship a live voice agent:

• [ ] Define explicit call states.


• [ ] Combine VAD with semantic end-of-turn detection.


• [ ] Add adaptive silence thresholds by question type.


• [ ] Make every streamed response cancellable.


• [ ] Drop stale TTS chunks after interruption.


• [ ] Classify barge-ins: ignore, duck, pause, or reset.


• [ ] Version user intent before tool calls.


• [ ] Require confirmation for irreversible actions.


• [ ] Track false cutoffs, repeat rate, and ignored interruptions.


• [ ] Review real call traces, not only aggregate dashboards.

FAQ

What is voice agent turn-taking?

Voice agent turn-taking is the system that decides when the user is speaking, when the user is done, when the AI should respond, and how the AI should behave if the user interrupts.

What is barge-in for AI voice agents?

Barge-in lets a user interrupt an AI voice agent while it is speaking. A good implementation stops or lowers the AI audio, listens to the user, and updates the conversation state without losing context.

Is latency the same as turn-taking?

No. Latency is about speed. Turn-taking is about timing and control. A low-latency agent can still feel bad if it cuts users off or ignores interruptions.

How long should a voice agent wait before responding?

It depends on the conversation. A yes/no answer may need only a short pause. A form field, address, or emotional explanation needs more patience. Use adaptive thresholds instead of one global silence value.

Should AI voice agents always allow interruption?

Usually yes for long spoken responses, but interruption should be classified. Backchannels like “mm-hmm” may only require audio ducking. Corrections and cancellation should pause or stop the response.

How do you test voice agent turn-taking?

Test with noisy audio, slow speakers, interruptions, corrections, accents, long tool calls, and users who change their mind mid-sentence. Measure false cutoffs, ignored interruptions, repeat rate, and barge-in stop time.