Google Cloud Tech: How voice AI agents control your browser

Code for this episode → https://g.dev/cloud/repositories

Gemini Live API docs → https://g.dev/cloud/gemini-live

CDP Page commands → https://g.dev/cloud/devtools-protocol



How does an AI voice agent control a real browser while keeping the conversation going?



This episode explains a design pattern for live voice agents: run suitable browser actions in the background, then use the next screenshot to observe the result. Keep read operations synchronous when the model needs their returned data to decide what happens next.



We cover the screenshot, reason, act loop, Chrome DevTools Protocol, tool classification through a callback, and server-side confirmation for consequential actions.



Chapters:

00:00 - Voice-Controlled Browser Demo

00:12 - Giving a Live Voice Agent Browser Control

00:46 - The Browser Control Loop

01:27 - From Model Decisions to Clicks: Chrome DevTools Protocol

01:57 - Why This Agent Uses Screenshots

02:43 - Keeping Slow Tools from Stopping the Conversation

03:45 - Async Actions vs. Synchronous Reads

04:11 - Classifying Tools with a Callback

04:36 - Human Confirmation and Server-Side Guardrails

05:25 - Next Episode: Long-Term Agent Memory





More resources:

CDP Input commands → https://g.dev/cloud/page-devtools-protocol

CDP Page commands → https://g.dev/cloud/input-devtools-protocol

ADK tool callbacks → https://g.dev/cloud/types-of-callbacks



Follow Annie online:

LinkedIn → https://goo.gle/annie-linkedin

X → https://goo.gle/annie-x



Watch more The Omni App → https://g.dev/cloud/modern-ai-app

🔔 Subscribe to Google Cloud Tech → https://goo.gle/GoogleCloudTech



#AIAgents #BrowserAutomation #ToolCalling #MultimodalAI



Speaker: Annie Wang

Products Mentioned: Gemini Live API, Agent Development Kit, Chrome