Aether-7B-5Attn: VIDRAFT's Fully Open-Source MoE Foundation Model with Five Heterogeneous Attention Mechanisms

TL;DR: Korean AI startup VIDRAFT has released Aether-7B-5Attn on Hugging Face — a fully open-source Mixture-of-Experts foundation model with 6.59B total parameters that integrates five distinct attention mechanisms arranged in a novel Latin-square layer structure. It ships under Apache 2.0 with full training data recipes, code, hyperparameters, logs, intermediate checkpoints, and evaluation code, making it one of the most transparently documented open releases to date.

What it is

Aether-7B-5Attn is a foundation language model developed by VIDRAFT (비드래프트), a Seoul-based Pre-AGI AI deep-tech company. The model is published on Hugging Face under the Apache 2.0 license and is part of VIDRAFT's broader four-layer integrated AGI system, which includes the Darwin model family, the AETHER architecture, the PROMETHEUS world model, and the HEPHAESTUS embedded AI stack.

Key release artifacts that are all publicly available:

• 
  Model weights (base and instruction-tuned variants)


• Complete training data recipe


• Full training code


• Hyperparameters and training logs


• Intermediate checkpoints


• Evaluation code


• Live inference demo

The project's stated philosophy is "true full-disclosure open source" — going significantly further than releases that publish weights alone.

How it works

Architecture: Mixture of Experts (MoE)

The model has 6.59 billion total parameters but activates only approximately 2.98 billion parameters per token, following the sparse MoE pattern where a router selects relevant expert sub-networks for each token. This keeps inference compute substantially lower than a dense model of equivalent total parameter count.

Five Heterogeneous Attention Mechanisms

The defining technical characteristic of this architecture is the simultaneous integration of five attention variants across the network:

1. 
   Full Attention — standard global self-attention


2. 
   Differential Attention — subtracts attention maps to suppress noise and improve focus


3. 
   Sliding Window Attention — local context windows for efficiency at longer sequences


4. 
   NSA-series Sparse Attention — structured sparsity patterns from the Native Sparse Attention family


5. 
   Hybrid Attention — combinations of the above

Rather than assigning all layers to a single attention type, VIDRAFT distributes these five mechanisms across the 49-layer network using a 7×7 Latin square arrangement. A Latin square ensures that each attention type appears exactly once in each row and column of the grid — preventing any single mechanism from clustering at a particular depth range and promoting balanced representational capacity throughout the network.

Training Data Composition

The model was trained on approximately 144.2 billion tokens with an intentionally multilingual, non-English-centric data mix:

• Mathematics: 37.8%
  


• Korean: 21.6%
  


• English: 21.6%
  


• (Remaining percentage covers other domains/languages)

This weighting reflects VIDRAFT's goal of building a foundation model with first-class Korean language capability alongside strong mathematical reasoning, rather than treating both as secondary to English.

Benchmarks & results

The source article does not report benchmark numbers specific to Aether-7B-5Attn itself. However, it cites the following public results for VIDRAFT's broader Darwin model family, which provides organizational context:

• 
  K-AI Leaderboard (Korea): Darwin series ranked #1 overall in 2026


• 
  GPQA Diamond: 90.9% score


• 
  Polaris Global New Drug Development Leaderboard: 14 category first-place rankings
  


• 
  Metacognition Leaderboard: #1 ranking
  


• 
  Cumulative Hugging Face downloads across the Darwin family: over 1,000,000 downloads
  

Aether-7B-5Attn's own evaluation results are available via the publicly released evaluation code on Hugging Face; engineers are encouraged to run benchmarks directly on their target tasks.

How to try it

The base model, instruction-tuned model, and a live demo are all publicly accessible via Hugging Face. To get started:

# Search for VIDRAFT's models on Hugging Face

huggingface-cli search vidraft

You can also browse directly at https://huggingface.co/vidraft or search for Aether-7B-5Attn in the Hugging Face model hub. The training code and evaluation scripts are available in the associated repositories.

VIDRAFT also develops VKAE (an inference acceleration engine) and VKUE (a lightweight runtime engine) as companion AI infrastructure — check the Hugging Face organization page for the latest availability of these tools.

FAQ

Q: Is this safe to use in commercial products?

A: Yes. The model is released under the Apache 2.0 license, which permits commercial use, modification, and redistribution provided you retain the license notice. Always verify the current license on the Hugging Face model card before deploying.

Q: Why five attention mechanisms instead of picking the best one?

A: Different attention mechanisms have different strengths at different network depths and context types. The 7×7 Latin square layer arrangement is designed to distribute all five mechanisms evenly across depth, so the model can leverage global context, local efficiency, and noise suppression properties simultaneously rather than making a single trade-off.

Q: Why is math weighted so heavily in the training data at 37.8%?

A: Heavy mathematical data is a well-documented strategy for improving general reasoning and instruction-following capability, not just numerical problem-solving. Combined with balanced Korean and English coverage, this mix targets strong reasoning across both languages.

Q: Is this related to VIDRAFT's quantum computing work?

A: VIDRAFT does research in quantum-assisted physical and life sciences, but Aether-7B-5Attn is a classical deep learning model. No quantum computing components are described in this release.

Originally reported by 텐센트 뉴스 (중국) (2026-07-20) — source article.