DeepSeek Just Patched the Core of Transformers - GPT & Gemini Look Old Now!

Author: Evolving AI - Bewertung: 3x - Views:27

DeepSeek just opened 2026 with one of the wildest AI architecture papers we’ve seen in years — a direct attack on the core weakness of huge Transformers: training instability and the GPU memory wall. In this video, we break down how their new Manifold-Constrained Hyper-Connections (mHC) widen the residual stream into multiple parallel paths, then force the mixing matrix to live inside the Birkhoff polytope so gradients stay bounded instead of exploding 3,000× or vanishing into noise. We’ll unpack what that actually means in plain English, why constraining A to a doubly stochastic matrix restores “identity-like” behavior, and how that lets DeepSeek scale models deeper without wrecking training dynamics. Then we go into the hardware side: how Hyper-Connections normally hammer GPU memory bandwidth, why HC can quadruple activation traffic across pipeline stages, and how DeepSeek stitched together fused TileLang kernels, recomputation, and DualPipe-style scheduling to slash memory movement and overlap communication with compute. We also look at the MoE results on a 27B-parameter model, where mHC delivers better stability, lower loss, and improved reasoning benchmarks like BBH and DROP with only ~6.7% overhead — turning a fragile idea into something that actually runs at scale. Finally, we zoom out to what this means for GPT, Gemini, Claude, and every lab that thought “just stack more layers” was the answer. Residuals gave us identity. Hyper-Connections gave us flexibility. mHC promises both — and if you want the real story behind the world’s fastest-moving AI breakthroughs, make sure to like and subscribe to Evolving AI for daily coverage.