LLM judge cost is the share of your eval bill spent grading agent output instead of producing it. To control it, run a 40-line offline pre-gate that triages every span with four deterministic rules and escalates only the uncertain tail to the expensive judge. On one trace this cut judge cost share from 50% to 16%.
LLM judge cost is the line item nobody puts on the FinOps dashboard. You add an LLM-as-judge to grade every agent span, you sleep better, and three weeks later the eval layer is quietly billing a third of what the agent itself costs. This post measures that share of your bill spent judging instead of doing, with a 40-line offline meter, and shows the one move that drops it from 50% to 16% on the same trace.
AI disclosure: I drafted this with an AI writing assistant. The tool, both fixtures, and every number below come from a real local run of
judge_gate.pyon Python 3.13.5, no network, no API key. I ran it, checked the exit codes, hashed the output twice to confirm it's deterministic, and edited every line myself before publishing.
Here's the sentence that set me off. Sattyam Jain wrote it on Dev.to on June 12, in a post arguing you should stop running an LLM judge on every agent call: "if your monitor exceeds ~20–25% of production cost, you built the wrong monitor." () Two days earlier the Linux Foundation announced its intent to launch the Tokenomics Foundation — open standards for AI cost management, because, in Jim Zemlin's words, "tokens have become the new unit of technology spend." ( gates the agent's action. The 's middle check, reused here as a free triage rule.)
Output schema. The output isn't even a JSON object — it's a raw string, or it's missing. → BAD.
200-with-empty-payload. Status says success, body is empty. The classic silent lie. → BAD.
Duplicate retry. This span's argument hash equals the previous span's. A byte-identical retry — the , a different tool.
Not a verdict on confidence fields. Honest limitation: my gate ignores a span's self-reported
confidence. One span in the fixture says confidence: 0.95, "no ambiguity" and still got escalated, because I refuse to trust a model's own confidence as a cheap signal — that's the kind of self-assessment that lies. If you trust yours, add a fifth rule. I didn't.Not a license to skip the judge. The judge gets the genuinely uncertain spans. The argument is against running it on the obvious ones, not against running it at all.
Run it on your own trace
Export 40–60 spans of a real agent run to JSONL with six fields per span (status, claimed_tool, tools_called, output, arg_hash, and prev_arg_hash carrying the previous span's hash so the duplicate-retry rule can fire), point judge_gate.py at it, and pass your real --judge-price and --prod-cost. If your judge cost share comes back under 10%, ignore me; your monitor's fine. If it comes back at 40%, you've found a line item.
One thing I genuinely don't know yet and would put real money on being argued in the comments: where the honest threshold is. Sattyam Jain says 20–25%. I shipped a default of 25%. But for a low-stakes summarizer, even 10% might be waste, and for an agent that moves money, maybe 40% is cheap. The budget is a --flag precisely because I don't think there's one right answer.
So I'll ask you: what's the judge cost share on a real eval pipeline you've shipped — and where would you set the budget before it counts as the wrong monitor?
I publish one runnable FinOps tool for AI agents at a time, with the real run log attached. Follow for the next number from the next trace — and drop your judge cost share in the comments, I read every one.
SOCIAL SHARE CARD GENERATOR