I Tested 300+ Models. Then I Killed the Benchmark.

Let It Break — part 1

Tags: #ai #llm #benchmark #postmortem

In May I ran a series called Agent Autopsy. Agents failing — broken packages, forgotten context, cron jobs dying silently — while I was learning what I didn't know I didn't know. Every failure got a post-mortem, because every failure felt like it deserved one.

That series didn't end with a finale. It ended when I stopped caring. Somewhere around part eight, broken things stopped feeling like emergencies. Call it tolerance. Call it resilience. Maybe I just got tired. Probably all three.

This is what came after: a series about what I kill on purpose, what I leave broken on purpose, and what that buys me. Nothing in this post failed. The benchmark worked perfectly, right up until the evening I killed it.

Back then I also wrote about . Days later: five brand new families debuted, none below 75%. Then: two models hit 90%, one for less than a penny.

Fifty percent, to a 75% floor, to 90% at sub-penny prices. In weeks.

When the punchline of every batch is "they're mostly fine and they're all cheap," the leaderboard has answered its own question. The last finding that actually helped anyone wasn't about model quality at all — . That efficiency is what kept the benchmark alive past its expiry date. When a zombie project costs pennies to feed and runs itself on cron, killing it requires noticing, not budgeting. Cheap automation doesn't just scale the good ideas.

And the harness had compromises I'd stopped seeing. The 400-token cap punished verbose-but-correct models. Pattern-matching scored format as much as competence. Fixing that meant more harness. The models were getting better faster than the harness could get fairer.

I built scaffolding to measure models. The models outgrew the scaffolding. That's not an engineering failure — that's the ecosystem working as advertised, and me billing myself weekly for refusing to notice.

What survives

The habit. I still test every model that interests me the day it drops, against tasks I actually care about. That reflex came from the benchmark and outlived it.

The findings. I know from data, not vibes: code quality does not equal agent capability, "write efficient code" prompts do nothing for most models, and free tiers charge you in debugging time.

The data, archived. If anyone ever genuinely needs it, it can be served through MCP in an afternoon — a query interface, not a leaderboard. No maintenance, no weekly re-runs, value on demand. The thin version of the same idea.

The harness does not survive. That's the right way around.

The rule

The weekend harness died on the whiteboard, where bad ideas are cheap. The benchmark died past 300 models, where they're not. Same disease, later diagnosis: "is anyone looking for this?" isn't a question you ask once at the start. You ask it every time you're about to maintain something.

Killing it cost one evening. Keeping it cost every week. I did that math embarrassingly late.

Next time I kill it at 50 models, not 300.

Tolerance, resilience, or just tired? This one was tired — and late. The next posts in this series cover the things I've left broken on purpose. Those are harder to defend. That's why they're worth writing.