Doubling Qwen3.6-27B on One RTX 3090: ollama llama.cpp + MTP, Lever by Lever (35.7 80.2 tok/s)

A reader on my , merged 2026-05-16), and that's where it worked for me. (There may well be an ik_llama path I missed — this is just what got it going on my box.)

Ollama's GGUF couldn't be reused. Qwen3.6 changed rope.dimension_sections from 3 to 4 elements; Ollama's stored blob still has the older 3-element layout, so llama.cpp refused it (expected 4, got 3). I grabbed a properly-converted GGUF instead (bartowski / a nextn-equipped MTP build) — a small heads-up if you're tempted to point llama.cpp at your existing Ollama blob.

With mainline llama.cpp, an MTP-equipped IQ4_XS GGUF, and --spec-draft-n-max 3, generation hit 80.2 tok/s.

Tuning MTP: more accepted drafts isn't more speed

The one knob that mattered for me was --spec-draft-n-max (how many tokens to draft ahead):

config	gen tok/s	draft acceptance
n-max 2	77.5	78.1%
n-max 3	80.2	70.3%
n-max 4	70.7	53.4%
n-max 3 + p-min 0.6	54.1	80.0%
n-max 3 + KV q8_0	74.6	64.5%

The counterintuitive bit: higher acceptance ≠ faster. Pushing p-min to 0.6 raised acceptance to 80% but dropped throughput to 54 — the extra rejected drafts cost more than they save. Plain f16 KV beat q8 KV too. n-max 3 with f16 KV was the sweet spot. (I also went looking for a "prefill-off" trick I'd heard about and couldn't find it as a flag in current llama.cpp — --spec-draft-n-max was the lever that actually moved the number for me.)

Honest caveats

Keeping these front and center, because they're the difference between a benchmark and a benchmark you can trust:

• 
  80.2 tok/s is this box's number (RTX 3090, WSL2). The originally-cited "~80" was a different setup; I reproduced ~80 honestly here.


• 
  Prefill numbers are noisy — my test prompt was short (~56 tokens), so I'm not headlining prefill. Generation tok/s is solid (±0.1).


• The bartowski Q4_K_M and Ollama's Q4_K_M are the same quantization family but different conversions (the rope change above), so they're not bit-identical weights. The model and quant family are matched; the conversion isn't.


• 
  Single GPU, single request. No batching or concurrency tested — that's a different question.


• One benchmarking trap that cost me time: llama-cli -n <N> is ignored under -no-cnv, so the model just generates until timeout (mine produced a 2 GB output file and looked like a 39-minute hang — it was runaway generation). Use llama-bench for token-exact non-MTP runs, and llama-server with n_predict for MTP.

Reproduce it

• 
  Hardware: RTX 3090 24 GB (Ampere, sm86), WSL2 Ubuntu 24.04, driver 591.74, nvcc 12.0.


• 
  ik_llama.cpp (commit bbe1a51): cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_NATIVE=ON
  


• 
  llama.cpp / mainline, has MTP (commit e3471b3): cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DBUILD_SHARED_LIBS=OFF
  


• 
  Models: bartowski/Qwen_Qwen3.6-27B-GGUF (Q4_K_M, IQ4_XS); a nextn/MTP-equipped Qwen3.6-27B-MTP-IQ4_XS GGUF for the speculative step.


• 
  Non-MTP bench: llama-bench -m <gguf> -p 56 -n 200 -ngl 99 -fa 1 -r 3
  


• 
  MTP run (the winner): llama-server -m Qwen3.6-27B-MTP-IQ4_XS.gguf -ngl 99 -fa on -c 4096 --spec-type draft-mtp --spec-draft-n-max 3, then POST /completion with n_predict: 200. Draft acceptance ≈ 70%.

Wrap-up

So the reader's nudge was a good one — Ollama really was leaving a clean ~2× on the table for this model on this card, and most of it is the MTP step. Ollama stays my default for everyday use (it's simple and it's what my tooling talks to); this build is the "I want every token/sec" setup. If you've gotten MTP working under ik_llama, or found the prefill trick, I'd genuinely like to hear how — that's the part I couldn't crack.

Vollständiger Original-Artikel

Den kompletten Beitrag mit allen Details direkt auf dev.to lesen.

↗ Original-Artikel auf dev.to lesen