Model migrations and dramatic agent speedups are recurring headlines. A single “2.2x faster” number cannot tell you whether your production workflow improves.
Build a paired workload from real, redacted tasks. Stratify by repository size, language, tool count, context size, and task type. Run old and new systems against the same pinned commit in isolated workspaces, randomize order, and repeat enough times to expose variance.
{
"task_id": "fix-017",
"system": "candidate",
"success": true,
"tests_passed": 184,
"wall_ms": 91200,
"model_cost_usd": 0.42,
"tool_calls": 19,
"human_rework_min": 6
}
Report median and tail latency, task success, regression count, human rework, tokens, retries, and total cost. Include timeouts and invalid patches in the denominator. Separate model latency from queue, tool, checkout, and test time.
Before launch, replay failures: duplicate tool results, network loss after a side effect, worker restart, context overflow, and cancellation. A faster system that repeats writes or cannot recover is not cheaper.
Use a predeclared acceptance rule, for example: no correctness regression above one percentage point, p95 at least 20% lower, and total cost per accepted task lower. Publish confidence intervals and the workload definition.
The benchmark should make a decision reproducible—not decorate one already made.