TurboQuant on a MacBook Pro: two findings the upstream discussion missed

Originally published at for Metal, validated on M5 Max, and took two PRs back to LLMKube to make TurboQuant first-class on the InferenceService CRD.

Why KV cache, why now

If you're running coding agents locally — single-model or architect+editor combos — the binding constraint isn't model weights. It's KV cache.

Weights you can quantize once, store on disk, and forget. KV cache is generated per token of context at inference time, sized by the model's depth and head dimensions, and held in working memory the entire session. A 35B-class model with flash-attn on uses roughly 256 KB of fp16 KV per token. That sounds small until you do the multiplication:

Context	fp16 KV
32K	~8 GB
64K	~16 GB
128K	~32 GB
256K	~64 GB
512K	~128 GB
1M	~256 GB

A 128 GB MacBook with flash-attn and mlock on can fit one 35B model at 128K with f16 KV, just barely. 256K doesn't fit. Co-resident two-model setups (architect + editor) don't fit at all past 64K.

Standard q8_0 quantization halves the KV footprint with sub-1% perplexity penalty. That gets you to 256K with a single model on the Mac.

TurboQuant () compresses further. Randomized Walsh-Hadamard transforms decorrelate KV blocks before scalar quantization, hitting ~3.25 bits per value (turbo3) or ~4.25 bits per value (turbo4) with attention-fidelity loss inside the noise floor of normal sampling variance.

Cache type	bits/value	Compression vs fp16	KV at 256K
f16	16.0	1.0×	~64 GB
q8_0	8.0	2.0×	~32 GB
turbo4	4.25	3.8×	~17 GB
turbo3	3.25	4.9×	~13 GB

Upstream discussion at and closes #350; #349 is next.

No perplexity numbers in this run. Throughput and memory ceilings only. The +1% perplexity penalty for turbo3 in the upstream discussion is on Qwen 3.5 — we'll re-run on Qwen 3.6 in a follow-up.

Single hardware sample. M5 Max only. Crossover point and prefill/decode split likely shift with memory bandwidth (614 GB/s on M5 Max) and GPU core count.

What we contributed back

• 
  .


• 
  .


• 
  Issues filed: .


• 
  Comment going to llama.cpp discussion #20969 with the M5 Max numbers and the prefill/decode split.

How to try it yourself

CODE Kopieren

# 1. Build TheTom's fork

git clone https://github.com/TheTom/llama-cpp-turboquant.git

cd llama-cpp-turboquant

git checkout feature/turboquant-kv-cache

cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release

cmake --build build -j



# 2. Run the bench (turbo3 and turbo4 separately to see the split)

./build/bin/llama-bench \

  -m /path/to/your/model.gguf \

  -ctk turbo3 -ctv turbo3 \

  -d 0 -d 32768 -d 131072 -d 262144 \

  -p 512 -n 128 -ngl 99 -fa 1 -r 3 -o md

Memory ceiling depends on your unified-memory budget; sub-64 GB Macs probably can't reach 256K with a 35B-class model at any cache type. M3 Pro/Max territory is more realistic for 13B models at 128K with turbo3.

For NVIDIA: or open an issue on defilantech/llmkube.

Vollständiger Original-Artikel

Den kompletten Beitrag mit allen Details direkt auf dev.to lesen.

↗ Original-Artikel auf dev.to lesen