I designed a 0.9B Mamba-2 / GLA hybrid LLM — the AI agents wrote the code. An honest build log.

Let me be clear about my role up front, because it matters: I didn't hand-write the code for this. I designed the system and directed it — the architecture, the decisions, the why, and the discipline of debugging it. The actual implementation was written by AI coding agents (Claude and Codex). I was the architect and the lead; they were the hands.

That collaboration is half the reason I'm writing this. The other half is that it's still a work in progress, and I'd rather show the honest version.

What it is

Helix v2 / Auralis — a ~0.9B-parameter hybrid language model, built from the tokenizer up (not a fine-tune, not an API wrapper):

• 
  28 layers, heterogeneous: 6× Mamba-2 (state-space) → 16× GLA (Gated Linear Attention) → 6× Sparse-Attention
  


• Pre-Norm (RMSNorm), RoPE, SwiGLU FFN


• 
  Tied 200k SentencePiece vocabulary, bilingual German/English
  


• d_model 1280, bf16

The reasoning behind the mix: cheap Mamba-2 (O(n)) at the bottom to move information, GLA in the middle, and a few precise Sparse-Attention layers on top where exact token-mixing actually matters — so most layers never pay the quadratic-attention cost.

Here's a cross-section of a single Mamba-2 mixer block — the state h replaces the quadratic attention matrix, giving linear-time inference and constant memory:

Honest status (work in progress)

It's at ~33k / 50k training steps:

• ✅ Trains stably; learns German and English fluently and keeps them separate


• ✅ Facts are anchored reasonably well in history & geography (measured, below)


• ⚠️ Science & translation are weaker; 0% code in the current data mix


• ❌ No instruction-following yet (no SFT) — ask a question, you get raw continuation, not an answer


• ⚠️ Greedy decoding is still rough

If you're looking for a chatbot to download, this isn't one yet. If you're here for the engineering, read on.

The part worth sharing: it was rarely the data

The most useful lesson wasn't architectural — it was how often my first explanation for a bad result was wrong, and how a careful process caught it.

At one point the model looked like it had regressed. My instinct (and that of two people I asked) was "the data must be bad." It wasn't. It was a stack of measurement problems:

1. 
   Learning rate too high for warm-start continued pretraining (carried over from a fresh-start schedule).


2. 
   Invalid baseline — comparing val-loss measured on two different validation sets.


3. 
   Wrong tokens/byte constant → ~33% inflated bits-per-byte. The model looked worse on paper than it was.


4. 
   Stochastic eval — nothing was re-seeded, so each evaluation drew different tokens. The "trend" was half real change, half sampling noise.


5. A wiki-only validation tail produced a fake cross-language gap of ~3.2 bits-per-byte; the real gap was ~1.04.

And the one that almost sent me chasing ghosts: "the model has no knowledge." Greedy decoding kept flip-flopping on simple facts. The conclusion "the facts aren't there" turned out to be wrong — I measured it properly with a contrastive margin (NLL(wrong) − NLL(correct) per token), and the facts were anchored. The flip-flop was a decoding artifact, not missing knowledge.

Here's the current evidence sheet — training curve, the metrics I actually trust, and an honest maturity grid of what works vs. what doesn't:

The takeaway I keep coming back to: before a bad number becomes "the data," check whether the number even measures what you think it does.

What helped

• 
  Deterministic eval (re-seed before every evaluation) — turned a noisy curve into a readable one


• A custom 200k tokenizer (the GPT-2 one was ~2× too inefficient for German)


• A two-stage data-cleaning pipeline, collecting data by knowledge profile rather than chasing total val-loss


• Treating knowledge, recall, and decoding behavior as separate things — conflating them cost me weeks

Licensing (precise on purpose)

• 
  Code: Apache-2.0 — fully open


• 
  Weights: OpenRAIL-M (responsible-use restrictions) — which means the weights are not OSI "open source" in the strict sense. I'd rather say that plainly than misuse the term.

What's next

The longer-term plan isn't just "make this one model bigger." It's a frozen universal base plus swappable DoRA/LoRA adapters — which is also why the large 200k vocabulary exists, and why its parameter cost gets cheaper as the base grows:

• Finish to 50k → SFT so it can follow instructions


• A small reproducible demo


• Then scaling — 1B is the foundation, not the goal (3B / 7B+), where the large 200k vocab finally earns its keep as its parameter share shrinks

Repo (critique very welcome):

👉 https://github.com/AuraIis/Helix

The most valuable part of this whole thing was having AI agents do the implementation while I stayed responsible for the decisions — and getting corrected, often, on my own assumptions.