They Put 7 Attention Mechanisms on a Latin Square. Then Removed Them One by One.

Since GPT, nearly every Transformer repeats the same attention mechanism at every layer. Forty-eight identical blocks, differing only in learned weights. Nobody tested that. It is a convention, not a conclusion. A paper out of VIDRAFT AI Research (arXiv:2609.20269, CC BY 4.0) tests it, and the interesting part is not the headline. The headline is... Weiterlesen