Why combine LiveKit and the Voice Agent API

WebRTC and AI are different problems with different best-in-class solutions:

• 
  LiveKit is the easiest way to ship production-grade real-time audio. SDKs for Web, iOS, Android, React Native, Flutter, and Unity. Built-in recording, simulcast, adaptive bitrate, and end-to-end encryption. A managed cloud and a self-hostable open-source server.


• 
  AssemblyAI’s Voice Agent API is the easiest way to ship a voice agent. One WebSocket gives you Universal-3 Pro Streaming for speech-to-text, an LLM, a TTS engine with 30+ voices, plus neural turn detection, barge-in, and tool calling — all server-side.

Use them together and you get multi-user voice rooms with a real AI agent inside, without writing a STT/LLM/TTS orchestration layer or building your own WebRTC stack.

How this differs from the LiveKit Agents framework

Architecture

The system has four layers:

Audio flows at 24 kHz mono PCM16 between the worker and the Voice Agent API. LiveKit’s native FFI resampler handles the conversion between WebRTC’s internal 48 kHz and the 24 kHz the API expects.

Prerequisites

• Python 3.10+


• An or a self-hosted livekit-server


• A LiveKit client to talk to the agent — the fastest path is the hosted :
  
  
  
  
  
  1. Open the playground.
  
  
  2. Paste your LIVEKIT_URL and a token. Generate a token from the LiveKit Cloud dashboard, set the room to voice-agent-demo and the identity to anything other than voice-agent.
  
  
  3. Click Connect , allow microphone access, and start talking.
  
  
  
  

  
  
  
  How it works
  

  
  
  

  The worker is one file (worker.py) and roughly 250 lines. Six steps do the actual work.

  
  

  
  
  
  1. Mint a LiveKit Token and Join the Room
  

  
  
  
  

  CODE Kopieren

   from livekit import api, rtc
  
  
  
  token = (
  
      api.AccessToken(LIVEKIT_API_KEY, LIVEKIT_API_SECRET)
  
      .with_identity("voice-agent")
  
      .with_grants(api.VideoGrants(
  
          room_join=True, room=ROOM_NAME,
  
          can_publish=True, can_subscribe=True,
  
      ))
  
      .to_jwt()
  
  )
  
  
  
  room = rtc.Room()
  
  await room.connect(LIVEKIT_URL, token)
  
  

  
  
  
  
  
  

  AccessToken builds a signed JWT with the grants the worker needs: subscribe to incoming audio, publish a reply track. room.connect() opens the WebRTC signaling and media path.

  
  
  

  
  
  
  2. Publish a Local Audio Track for the Agent’s Voice
  

  
  
  
  
  

  CODE Kopieren

   audio_source = rtc.AudioSource(sample_rate=24_000, num_channels=1)
  
  local_track = rtc.LocalAudioTrack.create_audio_track("agent-voice", audio_source)
  
  
  
  await room.local_participant.publish_track(
  
      local_track,
  
      rtc.TrackPublishOptions(source=rtc.TrackSource.SOURCE_MICROPHONE),
  
  )
  
  

  
  
  
  
  
  

  AudioSource is LiveKit’s pump for sending audio into a room. We configure it at 24 kHz mono — the Voice Agent API’s default format — so reply audio goes straight in without resampling.

  
  
  

  
  
  
  3. Subscribe to the User’s Audio Track
  

  
  
  
  
  

  CODE Kopieren

   @room.on("track_subscribed")
  
  def on_track_subscribed(track, publication, participant):
  
      if track.kind == rtc.TrackKind.KIND_AUDIO:
  
          asyncio.create_task(bridge_to_voice_agent(track))
  
  

  
  
  
  
  
  

  LiveKit emits track_subscribed when a remote participant publishes a track and it gets routed to us. We only care about audio.

  
  
  

  
  
  
  4. Forward Microphone Audio to the Voice Agent API
  

  
  
  
  
  

  CODE Kopieren

   stream = rtc.AudioStream.from_track(
  
      track=mic_track,
  
      sample_rate=24_000,    # ask LiveKit to resample to 24 kHz
  
      num_channels=1,
  
  )
  
  
  
  async for event in stream:
  
      pcm16_bytes = bytes(event.frame.data)
  
      await ws.send(json.dumps({
  
          "type": "input.audio",
  
          "audio": base64.b64encode(pcm16_bytes).decode("ascii"),
  
      }))
  
  

  
  
  
  
  
  

  AudioStream does the resampling. WebRTC carries audio at 48 kHz internally, but we ask for 24 kHz mono and the LiveKit FFI resampler handles the conversion. Each AudioFrame exposes data as a memoryview of int16 samples — base64-encode and ship as input.audio.

  
  
  

  
  
  
  5. Play the Agent’s Reply Back into the Room
  

  
  
  
  
  

  CODE Kopieren

   elif t == "reply.audio":
  
      pcm = base64.b64decode(event["data"])
  
      samples = len(pcm) // 2  # 2 bytes per int16, mono
  
      frame = rtc.AudioFrame(
  
          data=pcm,
  
          sample_rate=24_000,
  
          num_channels=1,
  
          samples_per_channel=samples,
  
      )
  
      await audio_source.capture_frame(frame)
  
  

  
  
  
  
  
  

  The agent streams reply.audio events as soon as the LLM begins generating. Each chunk is wrapped in an AudioFrame and pushed into the AudioSource, which queues it up to 1 second deep and drains at 24 kHz on its own clock.

  
  
  

  
  
  
  6. Handle Barge-In
  

  
  
  
  
  

  CODE Kopieren

   elif t == "input.speech.started":
  
      # User started talking; stop playback.
  
      audio_source.clear_queue()
  
  
  
  elif t == "reply.done":
  
      if event.get("status") == "interrupted":
  
          audio_source.clear_queue()
  
  

  
  
  
  
  
  

  AudioSource.clear_queue() immediately discards every queued frame so the user doesn’t hear stale agent audio after they’ve spoken over it.

  
  
  

  
  
  
  Tuning the agent
  

  
  
  

  
  
  
  Pick a Voice
  

  
  
  
  
  

  CODE Kopieren

   "output": {"voice": "james"}     # conversational US male
  
  "output": {"voice": "sophie"}    # clear UK female
  
  "output": {"voice": "diego"}     # Latin American Spanish
  
  "output": {"voice": "arjun"}     # Hindi/Hinglish
  
  

  
  
  
  
  
  

  See the .

  
  
  

  LiveKit ConnectError: invalid token. The JWT signature didn’t validate against the LIVEKIT_API_SECRET. Check that the URL, key, and secret all come from the same LiveKit project.

  
  
  

  Audio is choppy or robotic. Almost always the audio buffer running dry. Run the worker close to your network egress. Inside AudioSource(... queue_size_ms=1000) you have one second of headroom; raise it to 2000 if you see transient stalls.

  
  
  

  Audio sounds pitched up or down. Sample-rate mismatch. Both AudioSource and AudioStream.from_track must be configured at sample_rate=24_000, num_channels=1.

  
  
  

  Agent keeps interrupting itself. Browser clients with getUserMedia({ audio: { echoCancellation: true } }) handle this automatically. On custom mobile clients, make sure AEC is enabled on the capture side.

  
  
  

  The full troubleshooting guide is in the and the LiveKit Cloud pricing page.

  Vollständiger Original-Bericht

  Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.

  ↗ Original-Artikel auf dev.to lesen