Measure Your Agent's pass^k Before You Let It Run Overnight

Most teams shipping an agent know one number about it: how often it gets a task right. That number is measured with somebody watching. Your night queue has nobody watching, and it needs a different number. A recent benchmark put figures on the difference. Across 507 workflows that mutate stored state, twenty attempts each, the top model came out...