I Made My AI Models Argue, Then Let Hermes Be the Judge

The whole product is one question box. Everything interesting happens behind it, and the rest of this post is mostly pictures of that "behind."

Demo

Repo:

Try "Should a 3-person startup use microservices?" and open the dissent panel.

Local, one command (runs at $0 in offline mock mode, no key needed):

git clone https://github.com/ArqamWaheed/council && cd council && ./setup_hermes.sh && python server.py

Architecture, in pictures

I think the design is easiest to see, so here's the system as a sequence of images. Each caption is the explanation.

The bet. A hosted model and an on-device model sit on the same jury, swapped with a single --provider/--model flag, no code change. This model-agnosticism is the one Hermes property the whole project is built on.

The actual product. A confident single answer hides this; Council makes the disagreement the headline. Getting the clustering right here was subtle (see "What I learned" below).

The agentic learning loop, human-in-the-loop. Hermes proposes; you approve or dismiss. Approved rules persist client-side and ride along with the next convene call.

Interesting files:

• 
  hermes_run.py (the Hermes CLI driver every juror/judge call goes through)


• 
  run_council.py (orchestration + the deterministic judge + Hermes foreman + the --reflect loop)


• 
  skills/council/SKILL.md (the juror-weighting brain Hermes edits)


• 
  server.py (the /api/reflect + /api/learn endpoints)


• 
  index.html (the designed verdict UI with the foreman TTS readout and localStorage persistence).

Proof that Hermes is genuinely in the loop (subagent transcripts, skill diff, memory recall) is in

The judging brain is data, not a buried prompt. --learn and --reflect both edit this block, and the installed Hermes copy is kept in sync.

After a string of security questions, --learn appended a rule to upweight the local model on that topic (and synced the installed Hermes copy) because it had caught issues the hosted models missed:

python run_council.py --learn "Local Juror | security | 1.5"

On the next security question that juror's vote counts 1.5×, read straight back by the judge. Counterfactual: a static synthesis prompt can't get better; this does. (The before/after skill diff is in .

The foreman reads the verdict aloud. The verdict card has a "the foreman reads the verdict" button (browser SpeechSynthesis, $0); Hermes also ships native TTS via hermes setup tts. On-theme and memorable: a jury foreman announcing the decision.

The build itself was agent-run. I kept a memory.md the coding agent read before each task and updated after (so context stayed cheap), committed every increment with Conventional Commits, and built the verdict UI with the frontend-design skill, which is why the confidence dial and colour-coded juror chips read as designed, not default-template AI slop. The repo's AGENTS.md + commit history show the process, not just the result.

Why these models, and the concession. Two free OpenRouter models from different families (≥64K context, since Hermes rejects smaller at startup) plus a local Ollama juror. Two honest concessions: (1) free models are slower and three calls add latency (~10-20s/verdict); (2) the free tier is aggressively rate-limited, so I hit 429s constantly while building, and Council retries and, if a juror still won't answer, falls back (Hermes to direct API to deterministic stand-in) rather than crashing the verdict, which also means the demo runs fully offline at $0. For a once-a-decision tool, I'll take it. Cost: $0.

License. MIT. Fork it, add your own jurors.

What I learned (and what's next)

• 
  The disagreement is the product. A 2-1 split is more useful than a confident single answer, so the clustering that decides "who actually disagreed" has to be right. A small local model once wrote a vague position ("to facilitate efficient integration…") whose reasons clearly endorsed Postgres; the first version mis-filed it as a dissenter. The fix: when a juror's stated position is ambiguous, fall back to reading its reasons, and ignore options only mentioned in a comparison ("better than Mongo" isn't a vote for Mongo). Now agreeing jurors cluster together, and the split count is honest.


• 
  Grounded beats glib. Letting the agent propose its own weighting only works if the proposal is tied to real evidence; an ungrounded "reflect" just echoes whatever example is in the skill.


• Hermes' 64K-context floor caught a model that would've quietly underperformed.


• 
  A council should deliberate, not just vote. The round-2 debate above was the turning point: letting jurors read each other and reconsider means a juror that's genuinely persuaded moves the verdict, and you watch the confidence dial climb as a 2-1 split becomes unanimous. A one-shot vote can't do that.