On June 10, 2026, two completely unrelated systems hit quality gates that came back the wrong way. Neither system rationalized the verdict. Both honored it. That honesty is what makes the gate worth having.
The Thesis
A quality gate is only worth building if you will honor its verdict when the verdict is inconvenient. If you override a STOP without pre-registering a follow-up test, or fake a green check because a tool can't run, you're not adding rigor—you're adding theater. The gate becomes another notification that landed in your inbox yesterday.
On the same day, two teams in the Intent Solutions portfolio faced exactly that choice. The stories look nothing alike. The discipline they landed on was identical.
Story 1: Semantic-Flux—Honoring a Pre-Registered STOP on a $4 ML Signal Test
semantic-flux is building QCSS, a query-conditioned semantic search architecture. The patented core is a FiLM layer (feature-wise linear modulation) that lets the query modulate the document encoder's activations.
The project history matters because it shows why pre-registration exists. Earlier runs—v1, v2, v3—each produced results that got rationalized:
- v1: 4.79% nDCG FAIL. Someone overrode it. Reason: "It was undertrained." But no pre-registered test was written that could have disproven that excuse. The goalposts moved after seeing the number.
- v3: Used a pretrained MiniLM backbone, which is itself a trained retriever. The PASS measured the backbone's strength, not QCSS. The FiLM layer was never ablated away.
Three runs. No trustworthy verdict. Because every excuse closed the gate after the number landed.
So before v4 ran, the team locked a commitment into DECISIONS.md:
PROCEED to Phase 1 only if ALL three:
- film_lift (nDCG@10 with ON minus OFF) ≥ +0.03, 95% CI excludes 0
- scratch encoder beats BM25 floor (0.268)
- seed std ≤ 0.02
STOP if:
- film_lift ≤ +0.01 (conditioning inert)
- scratch encoder ≤ BM25 (can't beat 1990s baseline)
Timestamped. Before the number. The FiLM-ablation A/B (query-conditioning ON vs OFF on the same encoder) was the only way to isolate the patented mechanism.
On June 10, the v4 result landed (HF job 6a28d1c9, NVIDIA L4, 3 seeds, full NFCorpus = 3,633 docs):
- BM25 floor: 0.268
- BGE ceiling: 0.371
- Scratch encoder with FiLM-on: 0.0096
- Scratch encoder with FiLM-off: 0.0086
film_lift = +0.001—95% CI straddles zero
The FiLM weights never moved from 0.0002–0.0014 across every arm. The mechanism sat at identity.
STOP fired on two counts: film_lift inert AND scratch encoder 0.01 ≪ BM25 0.27.
The team honored it.
The honest read from DECISIONS.md: "This recipe does not work" because two things went wrong: (1) the from-scratch encoder learned almost nothing (trained on only 2,202 queries; MS MARCO has millions); (2) FiLM never trained—a zero-init / backbone-competition deadlock the team had flagged in advance. The result did not falsify QCSS—the mechanism never engaged—and did not support it either. The ~$4 signal test correctly said: do not spend the $2–5K Phase 1 yet.
Here's the transferable artifact. The team baked an anti-rationalization rule into the project: overriding a STOP requires a NEW DECISIONS.md entry that (i) names the specific confound believed to explain the result AND (ii) pre-registers a falsifying follow-up test. "It was undertrained" is not an admissible override without a pre-registered test that could disprove it.
That rule exists because v1's FAIL was overridden without (ii).
The result didn't make the business decision. But it made the decision honest and bounded: either retry with frozen backbone / non-zero FiLM init / higher FiLM learning rate + MS MARCO data (same locked gate, pre-registered falsifier), or halt empirics and file the structural claim. Either way, $4 stopped from turning into $5K of motivated spending.
Story 2: agent-governance-plane—Deleting a CI Gate Rather Than Faking It Green
agent-governance-plane is a TypeScript/Bun project—a policy and governance layer for agent sessions. On June 10 it shipped v0.1.46 and v0.1.47.
v0.1.46 delivered real infrastructure: PR #67 added credential injection into a sandbox plus a test that actually proves network isolation, not just assumes it. Shipped clean.
v0.1.47 (PR #68) set out to close two test-infra gaps:
- A Stryker mutation-testing gate—instrumentation of the two highest-risk files (src/policy/engine.ts, src/policy/dangerous.ts), a fail-closed mutation-gate.sh script, and the runner pinned to @stryker-mutator/, and it kills every gate downstream.
- Make it fake-green—
continue-on-error: trueor a script that always exits 0. A green checkmark that lies. The label says "Mutation testing passed" when mutation testing never ran.
Remove the check—keep the stryker.config.json and mutation-gate.sh as hash-pinned scaffolding, document the toolchain block in tests/TESTING.md, and file a tracked bead to re-wire it when a Bun-compatible runner exists.
They chose removal.
The commit message: "A permanently-red OR fake-green 'Mutation testing' check both violate the repo's honest-gate culture, so the CI job is removed."
The actual deliverable—the BDD acceptance layer—shipped unaffected. All the real hard gates passed: typecheck, biome lint, coverage-gate at 91.43%, claim-scan, doc-drift audit, harness verify, escape-scan (REFUSE=0, CHALLENGE=0).
The transferable point: a green checkmark is a claim. "Mutation testing passed" has to mean
{
"",
"@type": "BlogPosting",
"headline": "Honor the Gate When the Verdict Is Inconvenient",
"description": "A quality gate only matters if you honor its verdict. How pre-registration and honest-gate culture stopped two teams from faking green or rationalizing a STOP.",
"datePublished": "2026-06-10T10:00:00-05:00",
"dateModified": "2026-06-10T10:00:00-05:00",
"author": {
"@type": "Person",
"name": "Jeremy Longshore"
},
"publisher": {
"@type": "Organization",
"name": "Start AI Tools"
},
"url": "https://startaitools.com/posts/honor-the-gate-when-the-verdict-is-inconvenient/",
"keywords": "quality gates, CI gates, mutation testing, pre-registration, honest metrics, testing, ci-cd, ml-engineering, devops"
}
SOCIAL SHARE CARD GENERATOR