From Mixtral to Kimi K3: How Mixture-of-Experts Models Evolved

In this article, we'll discuss how Mixture-of-Experts models grew from a handful of experts to nearly 900 per layer, and the compression and stability mechanisms that keep such a sparse design trainable and affordable. Open-weight Mixture-of-Experts models have expanded at a remarkable pace: Mixtral had about 47 billion total parameters,...