Building a Voice-Controlled Local AI Agent with Whisper, Ollama & Gradio

Introduction

What if you could control your computer just by speaking to it — and have the

AI run entirely on your own machine, with no cloud, no API costs, and no data

leaving your device?

That's exactly what I built for my Mem0 internship assignment: a fully local

voice-controlled AI agent that transcribes speech, understands intent, and

executes real actions like writing code, creating files, and summarizing text

— all through a clean web UI.

What It Does

You speak (or upload audio). The agent:

1. Transcribes your speech using Whisper


2. Classifies your intent using a local LLM


3. Executes the action — writes code, creates files, summarizes text, or chats


4. Shows the full pipeline result in a Gradio UI


5. Asks for confirmation before writing any file (human-in-the-loop)

Example: Say "Write a Python retry decorator and save it to retry.py" —

the agent generates the code and saves it to your local output/ folder.

Architecture

Audio Input (mic / file upload)

│

▼

┌─────────────────┐

│ Whisper STT │ HuggingFace transformers (local) or Groq API fallback

└────────┬────────┘

│ transcript text

▼

┌──────────────────────┐

│ Intent Classifier │ Ollama (llama3.2) → returns structured JSON

└────────┬─────────────┘

│ [{"intent": "write_code", "params": {...}}]

▼

┌─────────────────────────────────┐

│ Agent Orchestrator │

│ • write_code → generate + save │

│ • create_file → write to disk │

│ • summarize → bullet points │

│ • general_chat → conversation │

└─────────────────┬───────────────┘

│

▼

Gradio Web UI

│

▼

output/ folder (sandboxed)

Models I Chose and Why

Speech-to-Text: Whisper (openai/whisper-base)

I used the HuggingFace transformers pipeline with openai/whisper-base.

It's ~74MB, runs on CPU without a GPU, auto-detects CUDA if available, and

has excellent accuracy for English commands. For machines where local

inference is too slow, I built in a fallback to Groq's Whisper API

(set STT_BACKEND=groq in .env) — near-instant transcription for free.

LLM: llama3.2 via Ollama

For intent classification and code generation I used Ollama running

llama3.2 locally. The key design decision was making the LLM return

structured JSON for intent routing:

[

  {

    "intent": "write_code",

    "params": {

      "filename": "retry.py",

      "language": "python",

      "description": "a retry decorator function"

    }

  }

]

This made tool routing deterministic and reliable. Temperature is set to

0.1 for classification (consistent) and 0.3 for code generation (slightly

creative but still focused).

Key Features

Compound Commands

The LLM returns a list of intents, not just one. So saying "Summarize

this and save it to summary.txt" triggers two intents in sequence —

summarize first, then create the file with the summary as content.

Human-in-the-Loop

Before any file is written to disk, the UI shows a confirmation prompt.

The user must click "Yes, proceed" or "Cancel". This can be turned off

with the auto-confirm checkbox for power users.

Safety Sandbox

All file writes are strictly limited to the output/ folder using path

sanitization. It's impossible to write outside this folder via voice

commands — path traversal attempts are silently stripped.

Graceful Degradation

• If Ollama is unreachable → falls back to keyword-based intent matching


• If local Whisper is too slow → falls back to Groq API


• If audio is unintelligible → shows a friendly error, no crash

Challenges I Faced

1. Gradio 6.0 Breaking Changes

Gradio 6 moved theme and css from gr.Blocks() to launch(), removed

show_download_button from gr.Audio(), and changed the Chatbot

component's message format. Each of these threw a TypeError at startup

and had to be fixed one by one by reading the changelog.

2. Python 3.13 venv Bug

python -m venv fails on Python 3.13 on Windows due to a pip bootstrap

issue. The fix was to skip the venv entirely and install packages directly

with pip install — perfectly fine for a development setup.

3. Ollama Port Conflict

On Windows, Ollama auto-starts as a background service after installation.

Running ollama serve manually throws a port conflict error. The solution:

don't run it manually — it's already running.

4. Structured JSON from LLM

Getting the LLM to reliably return valid JSON (no markdown fences, no

preamble) required careful prompt engineering. The system prompt explicitly

says "Respond ONLY with a valid JSON array — no markdown, no explanation"

and uses temperature: 0.1 to reduce hallucination.

What I'd Improve Next

• 
  Streaming responses — show LLM output token by token instead of
  waiting for the full response


• 
  Wake word detection — always-on listening with a trigger word


• 
  More tools — web search, calendar integration, running shell commands


• 
  Voice feedback — text-to-speech so the agent speaks its response back


• 
  Model benchmarking — compare Whisper tiny vs base vs large latency
  on the same hardware

Try It Yourself

The full source code is on GitHub:

👉 https://github.com/LostAlien96/voice-ai-agent

Requirements: Python 3.10+, Ollama, 8GB RAM. Setup takes about 10 minutes

following the README.

Conclusion

Building this taught me how quickly local AI has matured. Running

production-quality speech recognition and a capable LLM entirely on a

laptop — with no internet required after setup — would have seemed

impressive just two years ago. Today it takes an afternoon.

The most interesting design challenge wasn't the ML part — it was making

the system reliable: structured outputs, graceful fallbacks, safe file

sandboxing, and a UI that guides the user through confirmation steps.

That's where the real engineering work lives.