Mnemonic - local-first voice notes with Gemma 4 E4B

E2B and E4B are the only sizes with the audio encoder (~300M params). Both also ship with a ~150M vision encoder. The 26B and 31B are vision-only - no ears. "Use the biggest model that fits" is a non-starter for this product.

Between E2B and E4B, the deltas matter:

• 
  MMLU Pro 60.0 → 69.4 (+9.4). The difference between a model that fumbles unfamiliar technical vocabulary in voice notes and one that doesn't.


• 
  BBEH 21.9 → 33.1 (+11.2). Reasoning quality matters for self-correction ("actually, scratch that…") and for intent routing - one misclassification fires the wrong Shortcut.


• 
  CoVoST 33.47 → 35.54 and FLEURS 0.09 → 0.08. Modest audio-recognition wins.

At Q4_K_M the E4B GGUF is 4.98 GB (Hugging Face), plus audio and vision mmprojs (~1 GB combined). Co-resident with an IDE and browser on 16 GB.

One model, one pass - for both audio and vision

The conventional architecture for this product is two stages:

1. ASR (Whisper, Parakeet, etc.) → raw transcript


2. Text LLM → clean and structure

Mnemonic does both in one Gemma 4 forward pass. The audio goes into the model with a system prompt that says, in effect: "transcribe this and write it the way the speaker would write it themselves." Why it works better than two stages:

• One model in memory, one HTTP round-trip per recording. A 2-stage version means two model downloads, two warm processes, two failure modes.


• The cleaning prompt operates on the audio, not on a flat transcript. The model can hear pauses, hesitation, restarts - the difference between "I think" as filler and "I think" as opinion. A downstream LLM working from a transcript has already lost that.


• Lower end-to-end latency.

The same approach works for vision. A two-stage screenshot-with-voice product would be OCR (Tesseract, Apple Vision) → LLM merge. Mnemonic sends the WAV and the PNG to Gemma 4 in one multipart request, and the model produces a bullet that references both. The image isn't OCR'd in isolation - it's grounded by what the user said while taking it. Captions come out as "the panic the speaker mentioned in line 42" rather than generic "code editor with red error text."

Intent routing - a second narrow call, same model

Letting a voice note fire a macOS Shortcut took some thought. I didn't want to bolt on a tools/function-calling framework, an MCP server, or anything that added attack surface for a side-effect feature running on a user's machine.

What works is a second Gemma 4 call to the same llama-server, on the already-cleaned transcript, with one job - output a single JSON object:

{ "tool": "create-reminder", "input": "call Sarah at 3 PM" }

…or { "tool": "none" } if the transcript isn't a request. No tools registry, no plugin protocol - same model, same server.

Most of the work is around what doesn't fire:

• 
  Whitelist required. Mnemonic only runs Shortcuts named in allowed_shortcuts. A hallucinated name is refused before it reaches the OS.


• 
  No AppleScript, no shell interpolation. Input is piped to the Shortcut via stdin (shortcuts run NAME --stdin).


• 
  Undoable for 5 seconds. The tray menu shows Undo: <name> for the configured window. Click it to run a paired undo-<name> Shortcut.


• 
  Thought-dumps don't fire. "I was thinking about reminding Sarah, but maybe she already knows" → {tool: "none"}. Validated at 30/30 on hand-labelled transcripts, including 15 hedged/observational cases that must not fire (docs/spike/intent/PHASE-0-INTENT-FINDINGS.md).


• 
  Notes are the source of truth. A fire writes a ↳ Ran shortcut "<name>": <input> continuation line under the bullet. Greppable, auditable.

Cost is ~1.7s of warm latency per recording when enabled, off the user's critical path because the structuring queue is already async.

Implementation

Users run llama-server themselves, on loopback:

llama-server -hf unsloth/gemma-4-E4B-it-GGUF:Q4_K_M --port 5809 --mmproj-auto -ngl 99 -c 8192

The app posts one multipart request per recording (WAV + optional PNG + system prompt) to 127.0.0.1:5809, parses the JSON response, and appends a bullet. With intent routing on, a second JSON-mode call follows.

A few things that matter:

• 
  JSON mode + thinking mode. response_format: { type: "json_object" } for structure, chat_template_kwargs: { thinking: true } for chain-of-thought. The thinking tokens cost a few hundred ms but improve handling of technical terms.


• 
  The schema shrunk over three versions, then stayed put. v1 had seven fields (title, tags, summary, cleaned, actions, questions, entities). v1.5 had two. v2 has one: { cleaned: String }. Each prune was a UX win - the simpler the schema, the less the model felt the urge to narrate, summarize, or invent. The system prompt explicitly bans third-party narration ("the speaker", "the user", "the recording") and includes two calibration examples. v0.3's vision and intent work added new inputs and a new second call, but didn't grow that schema.


• 
  Failure is visible, not silent. llama-server unreachable, malformed JSON twice, silent audio - each produces a stub bullet with the timestamp and a redacted error. No recording is ever lost without a trace; the audio is preserved on disk, and the inbox queue means a job in flight survives a quit or crash.

What works well

• 
  Audio transcription and cleaning. Better than the Whisper + LLM pipeline I started with. Audio-grounded cleaning preserves intent across self-correction and hesitation in a way a downstream LLM on a flat transcript can't.


• 
  Vision captions are grounded. They answer "what was the speaker talking about" rather than describing everything in the image. Short, useful captions.


• 
  Intent routing is conservative. It refuses to fire more often than it fires. That's the right error direction for a feature that can run OS-level actions.


• 
  The queue makes the app feel instant. You stop waiting on the model. Tray returns to idle the moment you release the hotkey.


• 
  Everything is local. Loopback only. No keys to manage, no quota to worry about, no data leaves the machine.

Limits

• Processing is batched after the fact, not streamed. Fine for journaling; wouldn't work for live captioning.


• 16 GB unified memory is the floor. With a heavy IDE + browser open, memory pressure shows.


• I haven't tested non-English voice notes systematically. Gemma 4 is multilingual, but I work in English.


• No speaker diarization, no noise suppression. By design, most voice memos are solo.


• Intent routing requires building macOS Shortcuts by hand. Powerful if you set it up, but most users won't.


• Image OCR is good for screenshots, not for dense document scans. Short captions and inline text work well; multi-column papers don't.

UX

The whole interaction is one keystroke. Default is Ctrl+Option+Space hold-to-record (push-to-talk). Both the combo and the mode (hold or toggle) are configurable in ~/.config/mnemonic/config.toml, and the config hot-reloads - change a hotkey, save, the app re-registers without a restart. Tray transitions are perceptible within 100 ms.

The CLI ships a mnemonic doctor command with a green/red checklist for every common failure mode (mic permission, llama-server reachable, model loaded, mmproj loaded, paths writable). Brew install creates the mnemonic symlink on PATH automatically - no admin password prompt anywhere in the install path.

Built solo (well... not quite, Claude was involved). Source under MIT. Thanks to the Gemma team for shipping audio, vision, and chain-of-thought in a model that fits a laptop.