was more equivocal: the model leads on one of 12 coding-agent rows). But an independent harness came close to the opposite conclusion: a give its coding numbers a five-hour timeout, and up to 12 hours per run on PaperBench. The independent harness, VulcanBench, allowed . Qwen 3.8-Max lists at $2 and $6. Kimi K3 sits at $3 and $15.
Those prices tell you less than they used to, for a reason specific to reasoning models like Qwen: getting to a result costs thinking tokens. A model that spends most of its token allowance on reasoning can reach a token cap before it writes the answer, giving you an empty result indistinguishable from a total failure at the cost of a full run.
Artificial Analysis has : the harness logged a failure and nothing about why, and I had to add the distinction myself. When you do separate them, budget exhaustion turns out to dominate.
, Claude Opus 5's lowest-effort setting was its best, solving 20 of 23 tasks against 18 at high effort. The extra reasoning wasn’t useless: high effort returned the fewest wrong answers of any setting, one against three. It ran out of clock instead, and a timeout scores zero. Two of its three regressions were cutoffs on tasks that low effort solves, and given unlimited time on both it only ties its cheapest setting, at 3.1 times the cost.
That has a direct consequence for anyone building a routing ladder. The standard design escalates to more reasoning when a cheap attempt fails, on the assumption that the next rung is better and merely costs more. For a meaningful share of model and task combinations that assumption is wrong, and you pay the higher rung's price to escalate into a timeout or hitting a cap.
Who is already measuring this
Several groups have landed on cost per successful task independently in the last few months, which is the strongest signal it's becoming standard.
VulcanBench reports dollars per solved task as a headline column and , down from $1 per handled conversation. and bills only on end-to-end resolution.
What to change this week
Emit a failure reason on every agent run as a required field, with budget exhaustion, verifier failure and harness error as distinct values rather than one failure flag. Until you can separate a timeout from a wrong answer, your pass rate is measuring two things at once and you cannot tell which one to fix.
Compute cost per successful task per effort level, not just per model. Total spend including failed attempts, divided by tasks that passed your acceptance check. The ranking will not match the rate card, and the cheapest setting may well win.
Cap on tokens rather than wall clock unless latency is genuinely in your service level objective. A wall-clock cap scores your provider's serving speed as model quality.
Check the default effort setting on everything you have deployed. Qwen 3.8-Max runs at its highest reasoning setting when the effort field is unset, and its highest setting was its worst performer in independent testing. A team that never touches that parameter is running the configuration that costs the most per solved task.
SOCIAL SHARE CARD GENERATOR