Zum Hauptinhalt springen
tsecurity.de LIVE
Echtzeit-Radar & Feeds
Alle RSS Feeds
👥 Community & Social
Sichere ProgrammierungKI half beim Finden: iOS 27 schließt mehr als 100 Sicherheitslücken(21.09.2026 um 06:00 Uhr)
Sichere ProgrammierungWhat Is Rowhammer? How Can Repeated Memory Access Flip Bits in RAM?(21.09.2026 um 07:12 Uhr)
Sichere Programmierungnpm publish Ignores .gitignore: The .npmignore Override Rule(21.09.2026 um 07:15 Uhr)
Sichere ProgrammierungAphelion Editor - A free node-based video / VFX editor(21.09.2026 um 07:21 Uhr)
Sichere ProgrammierungGovernance Attack Surface Review: OKX(21.09.2026 um 07:31 Uhr)
Sichere ProgrammierungJSM Portal Request Create Property Panel Submit(21.09.2026 um 07:34 Uhr)
Reverse Engineeringsearch instructions assembly easy (X86,RISCV,AARCH64,etc)(20.09.2026 um 15:44 Uhr)
Sichere ProgrammierungKI half beim Finden: iOS 27 schließt mehr als 100 Sicherheitslücken(21.09.2026 um 06:00 Uhr)
Sichere ProgrammierungWhat Is Rowhammer? How Can Repeated Memory Access Flip Bits in RAM?(21.09.2026 um 07:12 Uhr)
Sichere Programmierungnpm publish Ignores .gitignore: The .npmignore Override Rule(21.09.2026 um 07:15 Uhr)
Sichere ProgrammierungAphelion Editor - A free node-based video / VFX editor(21.09.2026 um 07:21 Uhr)
Sichere ProgrammierungGovernance Attack Surface Review: OKX(21.09.2026 um 07:31 Uhr)
Sichere ProgrammierungJSM Portal Request Create Property Panel Submit(21.09.2026 um 07:34 Uhr)
Reverse Engineeringsearch instructions assembly easy (X86,RISCV,AARCH64,etc)(20.09.2026 um 15:44 Uhr)
Intelligence View
⚡ tsecurity.de Intelligence

Stop Leaving Findings in the Judge: The Ratchet That Turns Opinions Into Gates

Reagiere als Erste:r — dein Feedback zählt!

When a model-as-judge flags something real, most teams do the worst possible thing: they leave it in the judge. The finding lives on forever as a slow, metered, non-deterministic Tier 3 opinion — re-litigated on every run, at cost, with a different verdict each time. That's not an eval strategy. That's paying rent on a bug you already found.

The senior move is a ratchet: every recurring Tier 3 finding is a candidate to be promoted into a Tier 1 or Tier 2 check, where it becomes free, fast, deterministic, and able to block the run. This post is about how that ratchet works, and why it's the whole point of having a judge at all.

The independence axis, briefly

Before promotion makes sense, you have to rank evidence the right way. agent-eval ranks it on an independence axis — from independent to corruptible — not a cost axis:

  • Tier 1 — proof the agent can't forge. Valid JSON, the file exists, it compiled, tests passed, it finished inside the timeout, the output is non-empty.
  • Tier 2 — statistical signal vs a baseline the agent didn't author. Embedding similarity to the task spec, length and repetition profiles, whether the diff actually changed anything.
  • Tier 3 — model-as-judge. A shared-substrate opinion. A signal, never a verdict.

Two properties fall out of this and they drive everything below. Tier 1+2 are the real-time gate: deterministic, ~$0, fast, so they can sit in the hot path and block a run. Tier 3 is offline-only: metered, slow, non-deterministic, so it can't. And Tier 1+2 can legitimately run over agent trajectories, while Tier 3 cannot — a model judging another model's reasoning is circular, because judge and judged share a substrate and there's no independent ground truth. So Tier 3 only gets to inspect artifacts the judged agent didn't write.

The ratchet: promote findings down the tiers

Here's the pattern. Your judge keeps flagging the same class of failure — say, "the summary references a section that doesn't exist in the source." That's a real defect. But it's expensive to catch this way, and the verdict wobbles.

Ask: what unforgeable fact would have caught this? In this case, "every cited section header appears verbatim in the source" is a string-membership check. That's Tier 1. You just promoted a subjective opinion into a deterministic gate.

type Finding = {
  claim: string;        // the judge's recurring complaint
  sourceText: string;   // artifact the agent did NOT author
  citedSections: string[];
};

// Tier 1 promotion: the finding is now an unforgeable-proof check.
function citationsExist(f: Finding): { pass: boolean; missing: string[] } {
  const missing = f.citedSections.filter(
    (s) => !f.sourceText.includes(s)
  );
  return { pass: missing.length === 0, missing };
}

// Tier 2 promotion: when membership is too rigid, drop to a
// baseline-relative signal the agent didn't get to author.
function citationGroundedness(
  f: Finding,
  embed: (t: string) => number[],
  cos: (a: number[], b: number[]) => number
): number {
  const src = embed(f.sourceText);
  const scores = f.citedSections.map((s) => cos(embed(s), src));
  return scores.reduce((a, b) => a + b, 0) / (scores.length || 1);
}

citationsExist is Tier 1: it's a fact about text the agent can't forge. When exact membership is too brittle — paraphrased citations, translated sources — you don't jump back to the judge, you drop to Tier 2: citationGroundedness scores similarity against a baseline (the source) the agent didn't write. Still deterministic. Still ~$0. Still allowed in the hot path. Only if neither tier can express the property do you leave it with the judge.

Ship the 80%, meter the 20%

This is where the ratchet pays off. Most failures in production are not subtle: stale outputs, crashes, malformed JSON, hallucinated file paths, empty responses. Every one of those is catchable at Tier 1+2 alone — for free, in the hot path, blocking the run before a user ever sees it.

The judge is for the ~20% subjective tail: tone, whether an explanation is actually helpful, whether a refactor is tasteful. That work is real, but you run it offline, metered, and you label its output honestly: opinion, not evidence. The ratchet steadily shrinks that 20% over time, because every recurring judge complaint that can be expressed as a fact eventually gets promoted out.

This is the line that separates a real eval layer from an "LLM-as-judge gives you a 7/10" tool. A 7/10 is not a gate and it's not reproducible. A promoted Tier 1 check is both.

You can't promote what you can't see

The ratchet has a hard dependency: to promote a finding, you need the raw material — the exact source text, the resolved tool inputs, the actual output the agent produced. This is where the two halves of the workflow lock together.

agent-eval scores and gates the agent's output — the tiers, the drift checks, the hallucination checks above. But it can only score against unforgeable data if that data was captured faithfully. That's AgentLens: it captures the trace of how the agent got there — every model and tool step, the resolved inputs, the raw outputs. Two things follow. First, when a Tier 3 opinion recurs, you open the trace, find the exact artifact the agent didn't author (the source document, the tool response), and that becomes the baseline your new Tier 1/2 check scores against. Second, the trace is itself agent-didn't-author data, which is exactly what Tier 1+2 need to run against without becoming circular.

Without the trace, your judge is guessing and your gate has nothing trustworthy to grade. With it, every recurring opinion is one refactor away from becoming a free, deterministic check.

The takeaway

A model-as-judge is not the top of your eval stack — it's the intake queue for it. Its job is to surface recurring, real defects so you can promote them down the independence axis into checks that are cheaper, faster, and impossible for the agent to forge. agent-eval runs the tiers and the gate; AgentLens gives you the unforgeable trace to build the next gate from. Leave a finding in Tier 3 forever and you're not evaluating — you're just paying, slowly, to be reminded of a bug you could have gated on day one.

Ähnliche Beiträge
🔍 Verwandte News

Auch interessante Nachrichten Stop Leaving Findings in the Judge: The Ratchet That Turns Opinions Into Gates

Thematisch verwandte Begriffe: Stop, Leaving, Findings, Judge · 6 Treffer

Laden...

Videos werden geladen ...

Laden...

Beiträge werden geladen ...

Laden...

Videos werden geladen ...

Laden...

Beiträge werden geladen ...

Laden...

Videos werden geladen ...

Laden...

Beiträge werden geladen ...

Laden...

Videos werden geladen ...

Laden...

Beiträge werden geladen ...

Laden...

Videos werden geladen ...

Zum Aktualisieren ziehen
ZERO-DAY CVE-2026-94111 | Tencent BrowserSkill through 0.3.0 contains an authentication bypass vul…
Advisory →
TTS Reader • tsecurity.de Voice
tsecurity.de Icon
tsecurity.de App
Offline-Lesen, Eilmeldungen & 0ms Ladezeit

Installiere tsecurity.de direkt auf deinen Home-Bildschirm für das ultimative Vollbild-Magazinerlebnis ohne Browser-Leisten.

Nächster Beitrag
Themen-Radar & Intelligence Matrix
Echtzeit-Taxonomie nach Angriffsvektoren & Plattformen

tsecurity.de Live Threat Radar

🔴 LIVE RADAR
MONITORING
AKTIV
CVE-DATENBANK
LIVE
🔍
Community Radar & Live Chat
Sentinel Bot online • Live-Stream
Dein Cluster: Security Explorer
Match:
lädt…
Verbindung zum Community-Stream wird aufgebaut...
Bearbeitungsmodus — Senden überschreibt deine Nachricht
Community-Puls — was gerade passiert
lädt…
Aktivitäten deiner Analysten
lädt…
Neues Thema oder Eilmeldung einreichen

Reiche interessante Links, Zero-Days oder Debatten ein. Die Community entscheidet per Upvote über die Veröffentlichung.

Heiß diskutierte Einreichungen
🔖 Gespeicherte Artikel
📂 Keine gespeicherten Artikel vorhanden.
Zurück Ziehen Vor
Links: vorheriger Artikel Rechts: nächster Artikel unten: schließen
News NIS-2 Frühwarnung Tier-1 Intel ⏱️ 3 Min vor 10 Min
Artikeldaten werden geladen...

Zurück: vorheriger Vor: nächster
↗ Original-Quelle
Social Reaktionen Deine Reaktion zählt
Einstufung & Relevanz-Poll 0 Stimmen
In sozialen Netzwerken teilen 1-Klick