Is Your AI Agent Eval Set Actually Testing Anything?

Is your AI agent production-ready?

You shipped it with an eval set of five examples, all of them the demo you already knew worked. Two weeks later production is full of failures none of those five would catch. So you patch the prompt, the demo still passes, and you have no idea whether you fixed the class of problem or just that one screenshot.

That is the question this series opened with. In Part 1 we said production-ready is a bar you define before you build. The eval set is how you measure that bar, and it is worth more than the prompt it grades.

Why the eval set outlives everything else

Models change. You will swap the model, rewrite the system prompt, add a tool, switch providers to cut cost. Every one of those can silently break behavior that used to work. The only thing that tells you whether you regressed is the eval set. It survives every rewrite, because it encodes what "working" means independently of how you built it. A competitor can copy your prompt in an afternoon. They cannot copy two years of remembered failures.

Build it from real failures, not imagined ones

Most eval sets are weak because they are written from imagination at the start, when you know the least about how the agent fails. Invert it.

• Every incident becomes a permanent case. When the agent fails in the wild, the fix is not done until that exact failure is in the eval set. A bug you fixed and did not test is a bug you will ship again.
• Weight toward disqualifying failures. Part 1 asked you to name which failure modes are unacceptable. Your eval set should be heavy on those. That is where trust is won or lost.
• Include the boring middle. If 80 percent of real traffic is mundane, an eval set that is all edge cases measures a job nobody does. Match the distribution of reality.

Grade behavior, not exact strings

An agent is non-deterministic, so a golden test demanding one exact output string will flake, and you will start ignoring it. Open eval frameworks let you assert on the property you care about: did it refuse the unsafe request, call the right tool, stay under budget, avoid inventing a policy. An eval you learn to ignore is worse than none, because it costs attention and returns false comfort.

If this sounds like curating good test data, that is exactly what it is. An eval set is test data for judgment, and the expensive part is the same as it is for any realistic test data: deciding what a good case looks like. Do that thinking once, capture it, and every future model swap pays you back.

Part 3 takes this into reliability: once you can measure the agent, how do you measure something that will not give the same answer twice.

Your prompt is disposable; your eval set is the asset.