Your eval criteria are code. Version them like code.

A judge prompt is an implementation. The criterion it encodes is a contract, and ours rotted for three months before anyone noticed.

TL;DR: We version our prompts and our eval datasets in git. We never versioned the meaning of our eval criteria, the human-readable definition of what "complete" or "helpful" is supposed to mean. So when three people tweaked the "completeness" judge prompt over a quarter, the criterion drifted, kappa fell from about 0.70 to 0.55, and no diff explained why. We started treating each criterion as a versioned contract (a short definition, an owner, a date) kept separate from the judge prompt that implements it. Here is the shape that worked.

The silent rot

Our "completeness" criterion was a judge prompt, versioned in git like everything else. Over about three months, three different engineers edited it, each to fix a specific false positive they had hit. Every edit was reasonable on its own. The cumulative effect was that the criterion now meant something none of them had agreed to: stricter on multi-part answers, looser on follow-ups. Agreement with human labels slid from roughly 0.70 to 0.55. The prompt history was all there in git, but a diff of prompt text does not read as a definition. Nobody could answer the simple question: what is this criterion supposed to mean now.

Implementation versus contract

The judge prompt is the implementation: how we operationalize a criterion for a specific model. The contract is the human-readable definition: what the criterion means, who owns it, when it was last agreed. We had the implementation under version control and the contract nowhere. So the implementation could drift away from the (unwritten) contract one reasonable edit at a time, and nothing flagged it.

The contract shape that worked

criterion_id: completeness_v3

summary: answer addresses every sub-question in the user's message; a partial answer fails

owner: the eval lead for this surface

last_agreed: a dated review, re-confirmed when the prompt changes

judge_prompt_ref: git sha of the implementation

canonical_examples: two frozen passes, two frozen fails

from dataclasses import dataclass

from datetime import date



@dataclass

class Criterion:

    id: str

    summary: str            # the human-readable contract, not the judge prompt

    owner: str

    last_agreed: date

    judge_prompt_ref: str   # git sha of the implementation

    canonical_examples: list  # 2 pass, 2 fail, frozen as the meaning regression test

What changed in practice

Editing the judge prompt now forces you to touch the contract: bump last_agreed, update the summary if the meaning moved, and re-run the canonical examples. The canonical examples are the regression test for the criterion's meaning. If a prompt edit flips a frozen example, you did not fix a false positive, you redefined the criterion, and that goes through review as a contract change. Cheap edits that quietly shift meaning stopped being cheap.

The open question

The owner field is the part I am least sure how to scale. With thirty criteria and six engineers, half of them have no obvious owner, and an unowned criterion is exactly the one that rots. Assigning ownership without it turning into box-ticking is the unsolved part for us. If you have made criterion ownership stick on a real team, I want to hear how.

FAQ

Isn't the judge prompt enough documentation? No. A prompt is tuned for the model, not written for a human to read as a definition. The two drift apart, which is the whole problem.

How many criteria is too many? When you cannot name the owner of each from memory.

Which judge model? Genericize: a frontier model from a different family than the system under test. The cross-family part matters more than the exact name.