Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints

Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints Current agent benchmarks that rely on LLM judges are systematically unreliable—even on deterministic, replayed runs. This undermines almost every published leaderboard comparison. LLM Judge Endpoints Fail Basic Reliability:...