A small eval set built from your own traffic will answer questions a public benchmark cannot, because it is drawn from your distribution. What it cannot do is resolve a small difference — and the arithmetic for which differences it can resolve takes about five minutes.
Start from a decision, not a dataset
The failure mode of most first eval sets is that they were built to “measure quality”, which is not a question and therefore has no answer. Every useful eval set exists to support a specific decision that is about to be made:
- Can we ship this prompt change, or does it regress the hard cases?
- Is the cheaper model good enough for this step of the pipeline?
- Did last week’s provider-side model update break anything?
- Does the extraction pipeline handle scanned documents, or only clean ones?
Write the decision at the top of the file. It determines what goes in the set — a set for “can the cheap model do this” is weighted toward the boundary cases where it plausibly cannot — and it gives you grounds to refuse examples that do not bear on it. A set that measures everything measures nothing at a useful resolution.
What fifty examples can resolve
Suppose your grader gives each example a pass or a fail, and 45 of 50 pass. Your point estimate is 90%. The Wilson score interval — the one to use for small samples, rather than the normal approximation, which misbehaves near the boundaries — puts the 95% confidence interval at roughly [77%, 94%].
That is the number to internalise before building anything. A 50-example set that scores 90% is consistent with a true rate of 78% and with a true rate of 94%. It cannot tell you whether a change that moved the score from 88% to 92% did anything at all.
Two more results from the same arithmetic, both useful:
- The rule of three. If zero of n examples fail, the upper bound of the 95% interval on the failure rate is approximately
3/n. Fifty clean passes are consistent with a 6% real failure rate. A hundred are consistent with 3%. This is why “it passed all of them” is much weaker evidence than it feels like. - Sizing for a difference. Resolving a 5-percentage-point difference between two independent samples at conventional power needs several hundred examples per side, not fifty. If that is the question, either get the examples or accept that you are making a judgement call and say so.
from statsmodels.stats.proportion import proportion_confint
# Wilson interval — correct behaviour at small n and near 0 or 1.
lo, hi = proportion_confint(45, 50, alpha=0.05, method="wilson")
print(f"{45/50:.0%} 95% CI [{lo:.0%}, {hi:.0%}]") # 90% 95% CI [77%, 94%]
# Rule of three: zero failures in n trials.
for n in (30, 50, 100, 300):
print(n, f"failure rate could still be up to ~{3/n:.1%}")
None of this is an argument against a small set. It is an argument for knowing which claims it supports. Fifty examples are excellent at catching a change that breaks a whole category, and useless at ranking two models that are close — and the majority of real decisions are the first kind. only for what is left, and only after you have measured the judge’s agreement with your hand grades on the same examples. A judge that agrees with you 80% of the time is applying a 20% error rate to every result it produces, and that error is not random.
Two mechanical properties make an eval set much more useful and cost nothing at setup: run it against several models through one interface, so “what does the cheaper model score” is a loop over applies to your own sets, not only to public benchmarks.
SOCIAL SHARE CARD GENERATOR