I Built an AI Agent Orchestrator Where Gemma 4 Only Knows What You Teach It

Dashboard — 3-panel layout (skills · main · agents):

Chatting with a skill-scoped agent:

API Access panel — copy the curl command directly from the agent detail view:

Agents as MCP tools in the IDE:

How I Used Gemma 4

I used the gemma-4-31b-it model — the 31B dense instruction-tuned variant — via Google AI Studio through Spring AI's spring-ai-starter-model-google-genai.

Why the 31B dense, specifically:

The project enforces a hard constraint: agents must refuse anything outside their assigned skills and must do so with an exact phrase. This is a correctness requirement, not a quality preference — if the constraint breaks, the product doesn't work.

I tested smaller variants first. The 4B model followed the constraint most of the time, but would occasionally drift: offering "related" information outside its skills, or partially revealing the system prompt when directly asked. With the 31B dense, these failures essentially disappeared. The constraint held reliably across multi-turn conversations and adversarial inputs.

Two specific things the 31B unlocked that smaller models couldn't deliver consistently:

1. Long-context constraint adherence. A single agent's system prompt can carry 10,000+ tokens of skill content (multiple skill files, each with reference documents). The 31B model kept the opening STRICT CONSTRAINTS block in effect even with extensive context following it — smaller models would silently "forget" early instructions as context
   
   grew.




2. Role disambiguation. Many skill files written for Claude Code or agentic CLI tools contain dispatch instructions like "invoke subagent X" or "request tool Y." Injected directly into a system prompt, smaller models would sometimes output those templates literally. The 31B correctly understood the meta-instruction — "you are the agent being invoked, not the orchestrator invoking agents" — and applied the skill knowledge directly instead of outputting workflow templates.

Why not the 26B MoE? The MoE variant optimizes for throughput across concurrent requests. GemmaOrch is a single-tenant orchestrator where precision per response matters more

than requests-per-second. The dense model's full parameter activation per token is worth the inference cost for this use case.

Why not the 4B? For a general assistant or creative tool, the 4B is genuinely capable and would be my first choice to keep costs and latency low. But when "breaking the constraint" is a correctness failure — not just a quality degradation — the extra capacity of the 31B is justified.

The open-weights advantage: Gemma 4 is open. The application is architected so the model is an environment variable — swap AI Studio for a local Ollama instance and nothing else changes. For users with sensitive skill content (internal knowledge bases, proprietary processes), self-hosting is a real deployment path, not a future promise.

Switch from AI Studio to self-hosted in one line:

spring.ai.google.genai.chat.options.model=gemma-4-31b-it

Or run locally with Ollama:

ollama run gemma4:31b

Source: https://github.com/Bzaid94/gemma-agents-orchestrator.git · License: Apache 2.0