Announcing Long-Context Training on TPUs with MaxText Context Parallelism

Google Cloud's MaxText framework now supports long-context training up to 14 million tokens on Cloud TPUs by implementing four context-parallel strategies: All-gather, A2A, P2P, and USP. Powered by the Tokamax Splash Attention kernel, these strategies effectively manage attention compute and memory bottlenecks without materializing full key-value... Weiterlesen: Announcing Long-Context Training on TPUs with MaxText Context Paralle…