Beyond the "Brute Force Beauty": A Modular, Brain-Inspired LLM Architecture
— Notes on an attempt to disentangle "intelligence"
I. What's the Problem?
Current Transformer-based LLMs are powerful, but something feels fundamentally off:
Bloated: Hundreds of billions of parameters. Training costs tens of millions of dollars. Not accessible to ordinary people.
Black box: Change one parameter and you might affect grammar, semantics, facts, style… no one knows what's happening inside.
Context failure: No matter how large the window (128k, 200k), you get "lost in the middle." Long conversations lead to amnesia.
The root cause, in my view, is that all information is forced to "entangle" inside a single, giant parameter space — like mixing skin, flesh, and bones into a thick soup, then expecting the soup to grow into a human.
II. Where Did the Inspiration Come From?
- How the human brain works
Color is handled by area V4, shape by IT, local features (indentations, edges) by V2…
The prefrontal cortex (PFC) integrates information from these submodules, compares, eliminates, and decides.
Thinking and output are decoupled: You think "apple" in your head, but you can say "apple", "that red thing", or even "fruit". Thinking is abstract; output follows specific language rules.
- Extreme modularity in animals
New Caledonian crows: Dedicated tool‑use modules, lightweight and efficient.
Honeybees: Navigate by combining three independent modules: sun azimuth, landmarks, and sky polarization pattern.
Octopuses: The brain gives high‑level commands; each arm has its own "local intelligence."
"Synchronous Oscillation Binding" theory
The brain may use temporal synchronization of neuronal firing to "bind" different features (red + round + dimple → apple). Frequency itself becomes a semantic label; synchronisation equals communication.Decoupling in software engineering
A good complex system appears as a whole from the outside, but is highly decoupled on the inside. AI is no exception.
III. My Core Proposal
Goal
Design a modular, brain‑like, explainable, lightweight AI architecture to replace the current brute‑force entanglement paradigm of monolithic LLMs.
Overall Structure
text
┌─────────────────┐
│ Central Scheduler │ (analogous to PFC)
│ (Abstract LLM) │
└─────────┬───────────┘
│ task decomposition & integration
┌────────────┬──────────┼──────────┬────────────┐
▼ ▼ ▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐
│ Color │ │ Shape │ │ Local │ │ Memory │ │ ... │
│ Module │ │ Module │ │Feature │ │Retriever│ │ │
│(small NN)│ │(small NN)│ │ Module │ │(HippoRAG)│ │ │
└─────────┘ └─────────┘ └─────────┘ └─────────┘ └─────────┘
│ │ │ │
└────────────┴──────────┴───────────┘
│
┌─────▼─────┐
│Working │ (temporary scratchpad)
│Memory │
└───────────┘
Component Details
- Central Scheduler (PFC analogue)
Not a giant model, but a relatively lightweight yet highly abstract model (e.g., a few billion parameters).
Responsibilities:
Receive user input, decompose it into subtasks.
Invoke the appropriate sub‑modules (color, shape, memory, …).
Integrate results from sub‑modules, compare, eliminate, decide.
Finally produce an output that follows language norms.
- Sub‑modules (specialised processors)
Each sub‑module does one thing only:
Color module: recognises colour (could be a small CNN)
Shape module: recognises shape (small Transformer)
Local feature module: detects dimples, edges, etc.
Some modules could even be traditional programs (regex, math formulas).
Advantages: Single responsibility → explainable; lightweight → can be replaced/upgraded anytime.
- Memory System (solves the context window problem)
Working memory: temporary scratchpad for the current conversation/task. Small capacity, fast.
Long‑term memory: external, indexed knowledge base (inspired by HippoRAG, HawkinsDB). Stores huge amounts of facts, templates, experiences.
Flow: Scheduler first looks in working memory; if insufficient, queries long‑term memory and loads results back into working memory for processing.
Result: No fixed “context window” — as long as long‑term memory is large, the system can theoretically remember an infinite amount.
- Communication Protocol (synchronous oscillation binding)
This is the most elegant layer: outputs from different sub‑modules are not just thrown to the scheduler; they carry frequency tags.
Example: colour module outputs “red” oscillating at 40 Hz; shape module outputs “round” also at 40 Hz. When they synchronise, the scheduler knows these features belong to the same object.
Frequency itself becomes a semantic coordinate. Synchronisation = binding.
This could replace the expensive global self‑attention in Transformers.
IV. What Problems Does This Architecture Solve?
Current Problem How My Architecture Solves It
Bloated Total parameters = lightweight scheduler + several small modules + memory index. Far smaller than a hundred‑billion‑parameter monolithic model.
Black box Each module has a single function; failures can be localised. The scheduler’s decision process can be logged.
Context failure Replace fixed window with working + long‑term memory. Infinite context becomes possible.
Expensive training Modules can be trained/fine‑tuned independently. Some modules could even be traditional programs, costing nothing.
Hard to update knowledge Updating knowledge only requires modifying long‑term memory or fine‑tuning the relevant module, not retraining the whole model.
V. Open Questions (Next Steps)
How does the scheduler automatically decompose tasks?
Might need a “task grammar”, or let the scheduler learn to use tools (like Toolformer).
Concrete implementation of synchronous oscillation?
In a digital system, we could use learnable phase parameters. Some work already exists (SSA, GASPnet).
Standardised interfaces between modules?
All module outputs must be normalised (e.g., uniform vector dimension + frequency tag). Should this be hand‑designed or learned by the scheduler?
Efficiency of long‑term memory indexing?
HippoRAG uses knowledge graphs + PageRank, but real‑time retrieval might be slow. Need lighter solutions.
How to train the central scheduler?
It needs to learn “contrast memory information + output language norms”. Possibly multi‑task learning, or mimicking human prefrontal behaviour.
VI. Conclusion
This architecture is still a thought experiment, but it’s not built on thin air — every component has prototypes in the literature (CATS Net, MAP, HippoRAG, neural oscillation models…).
I believe the next breakthrough in AI won’t come from making models bigger, but from breaking “intelligence” into understandable, composable, and independently evolvable modules.
Just as good software must be decoupled, good AI should be decoupled too.
“Use the best algorithm to generate the best function for its purpose, then combine those best parts.”
If you are also interested in modular, brain‑inspired AI, let’s discuss. My next step is to build a prototype on a small‑scale task (e.g., multimodal image Q&A) to test feasibility.
April 2026, Suzhou
(continually updated)