VIDRAFT Releases Aether-7B-5Attn: A Fully Open-Source MoE LLM with Five Heterogeneous Attention Mechanisms

TL;DR: Korean AI startup VIDRAFT has released Aether-7B-5Attn, a 6.59B-parameter Mixture-of-Experts foundation model on Hugging Face under Apache-2.0 — shipping not just weights but training data recipes, full training code, hyperparameters, training logs, intermediate checkpoints, and evaluation code. If you care about reproducibility, sovereign AI, or heterogeneous attention research, this one is worth a close look.

What it is

Aether-7B-5Attn is an open-source foundation LLM developed by VIDRAFT (비드래프트), a Seoul-based Pre-AGI AI startup. It was released on Hugging Face under the Apache-2.0 license and is positioned as a truly open model — not just "open weights" — following the philosophy of projects like Allen AI's OLMo, Apertus, and LLM-jp.

Key facts from the release:

• 
  Total parameters: ~6.59 billion (MoE architecture)


• 
  Active parameters per token: ~2.98 billion (sparse activation for inference efficiency)


• 
  Attention layers: 49 transformer layers


• 
  Architecture novelty: Five heterogeneous attention mechanisms integrated into a single model


• 
  Training corpus: ~144.2 billion tokens


• 
  Data mix highlights: Mathematics (37.8%), Korean (21.6%), English (21.6%) — explicitly bilingual by design, not an English-first model retrofitted for Korean


• 
  License: Apache-2.0

Both a base model and an instruct-tuned model are available, along with a live demo, all through Hugging Face.

How it works

The central architectural idea is heterogeneous attention integration: rather than using a single attention mechanism throughout all layers, Aether-7B-5Attn combines five distinct attention types within one model:

1. 
   Full Attention — standard global self-attention


2. 
   Differential Attention — reduces attention noise by differencing two softmax attention maps


3. 
   Sliding Window Attention — limits each token's receptive field to a local window for efficiency


4. 
   NSA-family Sparse Attention — hardware-aware native sparse attention patterns


5. 
   Hybrid Attention — a mixed strategy combining the above approaches

The key engineering question when mixing attention types is: where do you put each type in the layer stack? Random or greedy assignment risks clustering certain mechanisms at certain depths, which can create uneven capability distributions across the network. VIDRAFT addresses this with a 7×7 Latin Square layout across the 49 layers — a combinatorial design that ensures no single attention type is overrepresented at any particular depth. Think of it as an experiment design technique borrowed from statistics, applied to layer-wise architecture decisions.

On top of that, the MoE backbone means only a subset of parameters (~2.98B) are active per forward pass, keeping inference costs manageable relative to the total parameter count.

The "fully open" commitment means the repository is designed so an external team can trace the complete lineage: data composition and tokenization → training code and configuration → logs → intermediate checkpoints → evaluation procedures. The goal is independent reproducibility, not just deployability.

Benchmarks & results

VIDRAFT has previously disclosed the following public benchmark results for their Darwin model family (the production-grade line sitting above the Aether research architecture):

• 
  K-AI Leaderboard: Overall #1 ranking in 2026


• 
  GPQA Diamond: 90.9%


• 
  Polaris Global Drug Discovery Leaderboard: 14 consecutive #1 rankings


• 
  Metacognition Leaderboard: #1


• 
  Hugging Face downloads: Darwin family has surpassed 1 million cumulative downloads
  

Specific benchmark numbers for Aether-7B-5Attn itself were not disclosed in the source article at time of publication. Given the full evaluation code is part of the open release, community-run evals on standard benchmarks (MMLU, HellaSwag, etc.) should emerge quickly.

How to try it

Both the base and instruct variants of Aether-7B-5Attn, plus a live interactive demo, are publicly available on Hugging Face. Search for VIDRAFT or Aether-7B-5Attn on .