Synod Update: Adding a Deterministic Safety Net (and Proving It Helps)

Quick update on Synod, the multi-agent code reviewer I've been building for

the Qwen Cloud hackathon.

What changed

I added a Semgrep pre-filter in front of the security agent. Before, every

finding came purely from the LLM reading the code and reasoning about it —

which works, but LLMs are stochastic. Same file, different run, sometimes a

different result.

Now Semgrep scans first with deterministic rules, and the security agent

validates and enriches those candidates instead of starting from zero every

time.

Did it actually help?

I was skeptical of my own change, so I benchmarked it properly instead of

assuming. Ran a single-agent baseline against the full council, with and

without the pre-filter, same vulnerable file, checked against known ground

truth:

MethodPrecisionRecallF1Single agent75%75%, but ranged 0–75% across runs75%Council, LLM-only75%same variance issue75%Council + Semgrep100%100%, every run100%

The interesting part wasn't the top-line numbers — it was that the

single-agent and LLM-only council both had real run-to-run variance.

Sometimes it caught everything, sometimes it missed half. That's not a

reviewer you can trust in CI.

Adding the deterministic scanner as a floor fixed that. It's not smarter,

it's just consistent — and consistency turned out to matter more than I

expected.

Also shipped

A GitHub webhook — open a PR, Synod reviews the diff and comments

directly, findings grouped by severity.

A small CLI, closer to how tools like Claude Code feel in the terminal,

for reviewing files or whole directories without touching the API

directly.

Repo's still open source: github.com/02NIN20/Synod

Built for the Global AI Hackathon Series with Qwen Cloud — Track 3: Agent

Society.