AI productivity gains are real, but smaller in production

A new meta-analysis on GenAI coding assistants landed on arXiv this month, and I think it is a useful cold shower for both sides of the argument.

The paper looked across 23 studies and found a statistically significant productivity gain from GenAI assistance in programming. Not magic. Not fake. A real effect.

But the effect was moderate, highly context-dependent, and smaller in open-source and enterprise settings than in controlled experiments. It also found no statistically significant learning gain.

That is basically the whole AI coding debate in one sentence: the tools help, but the demos are not the work.

And honestly, that feels right.

I use AI coding tools. They are useful. They save time on boilerplate, tests, refactors, library glue, CLI flags, and those annoying "I know what I want but do not want to type it" moments.

But if you spend enough time around real systems, you also know why the big productivity promises feel slippery. Programming in production is not just generating code. It is understanding weird constraints, changing old systems without breaking them, negotiating ownership, reading logs, dealing with flaky tests, waiting for reviews, managing releases, and discovering that the simple change touches a tax rule from 2019.

The assistant can help with parts of that.

It does not delete the rest.