Which LLM should I actually code with? I built a small benchmark to find out

A small, self-run coding benchmark: 3 models on 14 problems across 3 languages, scored on pass@k, cost, and speed. Last run 11 July 2026.

peculiarengineer.com

I kept going back and forth on which model to reach for in my actual day job. Every "which LLM is best at code" thread turns into vibes and screenshots, and none of it answered the question I had, which is which one to open when I have real work in the languages I use. So I stopped guessing and built a small benchmark to settle it for myself.

It is deliberately small. 14 problems across Python, C#, and Bash, three models, three attempts each at temperature 0.7 with a 10 second timeout. Every attempt runs in a sandboxed Docker container and gets scored on pass@k, cost, and latency. It is not an authoritative ranking and I am not pretending it splits hairs. It is enough to show the shape.

The thing that surprised me

Accuracy is not the differentiator anymore. All three models solved every problem they were allowed to answer, 100% pass@3. If I only looked at pass rates I would have learned nothing, because they all pass.

The catch hides in "allowed to answer." One model got content filtered out of four problems, so its perfect score covers 10 of the 14, not the whole set. A perfect score on the problems you answered and a perfect score on the whole bench are not the same result, which is why the leaderboard sorts on coverage first.

Where they actually differ

If they all pass, the decision comes down to what you pay and how long you wait. That is where the spread lives.

• Cost was close, about 1.3x from cheapest to priciest across the suite.


• Latency was not close. 6.6x between the fastest and the slowest.


• The cheapest run that also covered all 14 problems came in around $0.028 per solved problem.

So the honest summary is boring in the best way. Pick on speed and price, because accuracy already agrees.

The gotcha worth writing down

One result is not like the others. One model was quick and cheap on Python and C#, a second or two per problem, then fell off a cliff on Bash at 90 to 168 seconds per problem. Four slow Bash problems dragged its average latency up to something that makes it look slow overall when it is really fine everywhere except Bash. If your day is mostly shell, that matters a lot. If it is Python, you would never notice.

That is why a single "which model is fastest" number is a lie. Fastest at what, in which language, is the only version of the question worth asking.

What I took away

For my work it came down to speed and cost per language. The benchmark did the one job I wanted. It turned a running argument in my head into a few numbers I can re-run whenever the models change.