Apple Silicon LLM Inference Optimization: The Complete Guide to Maximum Performance

TL;DR: MLX is 20-87% faster than llama.cpp for generation on Apple Silicon (under 14B params). Use Ollama 0.19+ with the MLX backend for 93% faster decode with zero config. Q4_K_M is the sweet spot quantization (3.3% quality loss, 75% size reduction). On a 32GB Mac, top picks include Qwen 3.5 9B (daily driver), DeepSeek R1 Distill 14B (reasoning), Qwen 3.5 35B-A3B (MoE), and OpenAI gpt-oss-20b — but the "best" model depends on your use case, context length needs, and quality tolerance. Memory bandwidth is your bottleneck — not compute, not VRAM, not GPU cores. This guide covers every optimization that matters.

I run a 32GB M4 Mac Mini as my local inference box. After weeks of benchmarking different engines, quantization levels, models, and optimization techniques, I've compiled everything into one reference. The Apple Silicon inference ecosystem has matured dramatically in 2026 — MLX is no longer experimental, Ollama ships an MLX backend, and vLLM has two competing Apple Silicon ports. The performance gap between local and cloud is narrowing fast for mid-size models.

This guide is specifically about optimization — squeezing maximum tokens per second at maximum quality from Apple Silicon hardware. If you need a broader overview of inference tools and hardware, see my local LLM inference guide and Mac Mini buying guide.

The Bandwidth Bottleneck

LLM token generation is memory-bandwidth-bound, not compute-bound. Every generated token requires reading the entire model's weights from memory once. This means your tokens-per-second ceiling is a simple formula:

Max tok/s = Memory Bandwidth (GB/s) ÷ Model Size in Memory (GB)

Applied to M4-series chips:

Real-world numbers hit 60-80% of theoretical due to KV cache reads, attention computation, and kernel overhead. But the relationship holds: quantization is a direct multiplier. Going from FP16 to Q4 gives you 4x throughput because you move 4x less data per token.

This is why buying more GPU cores barely helps. The M4 Pro's 16 GPU cores vs the base M4's 10 cores is a 60% increase in compute — but its 2.3x increase in memory bandwidth is what actually drives the 2x+ inference speedup.

MLX vs llama.cpp vs Ollama: Benchmarks

The Apple Silicon inference landscape shifted in 2026. MLX went from experimental to the fastest option for most workloads.

Generation Speed (decode tok/s)

M4 Max (128GB):

Source: Groundy benchmarks

Key pattern: MLX leads by 20-87% for models under ~14B. The advantage collapses at 27B+ where memory bandwidth saturates and both frameworks hit the same ceiling.

Ollama 0.19 MLX Backend vs llama.cpp Backend

Ollama 0.19 added an MLX backend that auto-activates on Macs with 32GB+ RAM:

Model: Qwen3.5-35B-A3B on M4 Max 64GB. Source: DEV Community

To enable: update to Ollama 0.19+ and set the environment variable:

export OLLAMA_MLX=1

ollama run qwen3:14b

The MLX backend requires 32GB+ unified memory. Base M4 models with 8-16GB will not activate it.

The Prefill Problem

MLX has a critical weakness: it performs full prefill before emitting any tokens. This means time-to-first-token (TTFT) rises linearly with input length.

Benchmark at 8.5K context (M1 Max, Qwen3.5-35B-A3B):

Source: famstack.dev

The UI reported 51 tok/s for MLX generation, but effective wall-clock throughput was only 3 tok/s because 94% of time was spent prefilling.

When to Use Which

Quantization Deep Dive

K-Quants: Why They Exist

K-quants (Q4_K_M, Q5_K_M, Q6_K) replaced legacy quantization (Q4_0, Q4_1) in llama.cpp. The difference is architectural:

K-quants deliver 3-4x less perplexity increase at the same file size. The llama.cpp maintainers now recommend Q3_K_M over Q4_0 — the 3-bit K-quant is better than the 4-bit legacy format.

The S/M/L suffixes control metadata overhead: S (smaller, faster), M (balanced — use this), L (more metadata, best reconstruction).

Quality Comparison by Quantization Level

Measured on Llama-3.1-8B-Instruct, WikiText-2 perplexity:

Source: arXiv 2601.14277

Practical guide:

• 
  Have the RAM? Use Q5_K_M — 1% quality loss is the best tradeoff


• 
  Tight on RAM? Use Q4_K_M — 3% loss is imperceptible for chat/coding


• 
  Want lossless? Q8_0 is 0.14% loss — essentially identical to FP16


• 
  Desperate for space? Q3_K_M is the floor — below this, quality falls off a cliff

GGUF vs MLX Format vs AWQ vs GPTQ

GGUF is the only viable format on Apple Silicon for llama.cpp/Ollama. MLX has its own format (safetensors) that's 20-30% more memory-efficient due to zero-copy unified memory. MLX can read GGUF files, but only Q4_0, Q4_1, and Q8_0 — all other quants get cast to FP16, defeating the purpose.

Bottom line: Use GGUF for Ollama/llama.cpp. Use MLX-native models (from mlx-community on HuggingFace) for mlx-lm. Ignore AWQ and GPTQ entirely — they're NVIDIA-only.

RAM Requirements by Model Size

The formula: Model RAM (GB) = Parameters × Bits_per_weight ÷ 8. Keep model weights under 60% of total unified memory to leave room for macOS, KV cache, and the inference runtime.

What Fits at Each RAM Tier

What Fits on 32GB

Usable memory budget: ~20GB for model weights (60% rule).

My recommended starting points for 32GB (but see Choosing a Model below):

• 
  Primary: Qwen 3.5 9B Q5_K_M — general use, coding, writing, multimodal


• 
  Reasoning: DeepSeek R1 Distill 14B Q4_K_M — chain-of-thought problems


• 
  Speed: Llama 3.1 8B Q4_K_M — quick tasks, agent chains


• 
  Maximum capability: Qwen 3.5 35B-A3B Q4 — MoE, 35B quality at near-8B speed


• 
  Alternative: OpenAI gpt-oss-20b Q4_K_M — strong general purpose, Apache 2.0

Choosing a Model: It Depends

There is no universally "best" model. The right choice depends on factors that vary by person and use case. Before defaulting to whatever is trending on HuggingFace, ask yourself:

What matters most to you?

The model landscape moves fast. The Qwen 3.5 family launched in February 2026 and already has models with 5M+ downloads. Gemma 4 arrived in March 2026 with a compelling MoE variant. OpenAI released their first open-weight model (gpt-oss-20b) in August 2025. By the time you read this, newer options may exist.

How to evaluate a new model for your setup:

1. Check HuggingFace for downloads, community ratings, and GGUF availability


2. Verify it fits your RAM budget using the RAM table above — model weights under 60% of total memory


3. Look for Unsloth Dynamic 2.0 GGUFs or mlx-community conversions for best Apple Silicon performance


4. Test on YOUR use cases — benchmark scores do not always correlate with subjective quality for your specific tasks

MoE vs Dense Models

The critical misconception: MoE does NOT save memory. All expert weights must reside in RAM — the router needs access to every expert to make routing decisions. MoE saves compute, not memory.

What MoE does give you on Apple Silicon:

• 
  Faster token generation — fewer active parameters per token means less computation


• 
  Higher quality per RAM dollar — a 30B MoE with 3B active params gives you 30B-class knowledge at near-3B speed


• 
  Same memory footprint — Mixtral 8x7B (46B total) needs ~26GB at Q4, same as a dense 46B model

MoE Models on 32GB Mac

The MoE landscape has expanded significantly in early 2026. Two standouts for constrained hardware:

• 
  Qwen 3.5 35B-A3B (3.5M downloads, 1.3K likes) — successor to Qwen 3 30B-A3B with multimodal vision support. Only 3.3B active parameters means near-8B generation speed despite 35B total knowledge. On M4 Max, expect 64-92 tok/s.


• 
  Gemma 4 26B-A4B (1.3M downloads) — Google's MoE entry with 4B active params. Fits more comfortably in 32GB than Qwen 3.5 35B-A3B (~15GB vs ~19GB), leaving more room for KV cache and context.

The memory bandwidth formula explains why MoE helps: you still read all weights from memory, but the routing + expert computation is cheaper, so the gap between theoretical max and real-world performance is smaller.

vLLM on Apple Silicon

Two competing projects bring vLLM's serving capabilities to Mac:

vllm-mlx

Built on MLX. The more feature-complete option.

pip install vllm-mlx

vllm-mlx serve mlx-community/Qwen3-14B-4bit

• OpenAI + Anthropic Messages API compatibility


• MCP tool calling support


• Multimodal (text, vision, audio, embeddings)


• Continuous batching — 2-3.4x throughput with concurrent requests


• 21-87% faster than llama.cpp

Source: github.com/waybarrios/vllm-mlx

vllm-metal

Official vLLM community plugin with native Metal kernels.

curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm-metal/main/install.sh | bash

• PagedAttention with Metal kernels


• Full vLLM scheduler integration


• Text-only (no multimodal)


• No published benchmarks yet

Source: github.com/vllm-project/vllm-metal

Use vllm-mlx if you need multi-user serving from a Mac. Use Ollama for single-user inference.

When your Mac's memory ceiling limits model size or concurrent users, Railway deploys GPU inference servers (vLLM, Ollama) from GitHub with autoscaling — no DevOps required.

Optimization Techniques

1. KV Cache Quantization

The KV cache stores attention states and grows linearly with context length. On a 14B model at 32K context, it can consume 4-8GB — a huge chunk of your 32GB budget.

llama.cpp supports KV cache quantization via flags:

llama-server -m model.gguf --cache-type-k q8_0 --cache-type-v q8_0

This is free performance. Q8 KV cache halves memory usage with negligible quality loss, letting you either run larger models or longer contexts.

2. Speculative Decoding

A small "draft" model generates candidate tokens; the large "target" model verifies them in a single forward pass. Output is mathematically identical — zero quality loss.

Apple's own Recurrent Drafter research achieved 2.3x speedup on Metal. Typical acceptance rate is 60-80%.

# llama.cpp speculative decoding

llama-server -m qwen3-14b-q4.gguf --draft qwen3-0.6b-q8.gguf

3. Flash Attention on Metal

Available through community implementations:

• 
  Metal FlashAttention 2.0 — 43-120% faster attention computation


• 
  mlx-mfa — Flash Attention for MLX


• 
  MLX built-in — Uses optimized Metal attention kernels (not branded Flash Attention, but similar benefits)

The main benefit: reducing attention memory from O(n^2) to O(n) in sequence length. Critical for long-context inference on memory-constrained machines.

4. Context Length vs Speed

Longer context = dramatically slower generation. At 40-50K tokens, expect 10x slower performance than short context.

Practical limits on 32GB (model weight budget ~20GB):

With Q8 KV cache quantization, roughly double these context limits.

Best Models for Every Mac Configuration

These tables show strong options per tier — not the single "best" model. See Choosing a Model for how to decide.

M4 Base (16-32GB) — 120 GB/s

M4 Pro (24-48GB) — 273 GB/s

M4 Max (64-128GB) — 546 GB/s

The M4 Chip Architecture

Does the Neural Engine Help?

Not yet — but research is progressing. The 16-core Neural Engine (38 TOPS) is designed for vision, speech, and sensor data — not the large matrix operations in LLM inference. MLX does not use the ANE. Neither does llama.cpp. CoreML can theoretically route to ANE but imposes model size limits that make it impractical for 7B+ models.

However, the Orion project (March 2026) demonstrated the first open system for direct ANE programming, bypassing CoreML entirely via private APIs. It achieved 170+ tok/s for GPT-2 124M inference on M4 Max and documented 20 ANE hardware constraints (14 previously unknown). While not yet practical for large models, Orion shows the ANE has untapped potential for LLM workloads.

For now, the GPU is your inference engine. All production frameworks use Metal shaders on the GPU cores.

Note on M5: Apple's M5 chip introduced Neural Accelerators inside each GPU core specifically for matrix multiplication, providing up to 4x speedup for time-to-first-token. This is different from the standalone Neural Engine — it's integrated into the GPU pipeline. M5 memory bandwidth is 153 GB/s (28% higher than M4), yielding 19-27% faster token generation. MLX supports these accelerators natively.

Unsloth: Training Tool, Quantization Game-Changer

Unsloth is primarily a fine-tuning tool — it makes QLoRA training 2-5x faster with 50-70% less VRAM. It does NOT run inference. For inference, use Ollama/llama.cpp/MLX.

Apple Silicon training status: Not yet supported natively. Requires NVIDIA GPUs (CUDA). MLX training is listed as "coming soon."

Why Unsloth matters for you anyway: Dynamic 2.0 GGUFs.

Unsloth Dynamic 2.0 is an advanced quantization method that:

• Analyzes each layer individually and picks the optimal quantization type per layer


• Uses a calibration dataset of 1.5M+ tokens for conversational optimization


• Reduces KL divergence by 5-8% compared to standard quantization at the same bit level


• The resulting GGUFs run on any inference engine (llama.cpp, Ollama, LM Studio)

Download Unsloth Dynamic 2.0 GGUFs from HuggingFace for the best quality-at-size. You don't need Unsloth installed — just the model files. Top Unsloth GGUFs for Apple Silicon:

Sources

Research Papers

Benchmarks and Guides

Tools and Frameworks

• 
  MLX Framework (Apple) — Apple's native ML framework for Apple Silicon


• 
  llama.cpp — CPU/GPU inference engine with Metal backend


• 
  Ollama — One-command local inference with MLX backend


• 
  vllm-mlx — vLLM serving on Apple Silicon via MLX


• 
  vllm-metal — Official vLLM Metal plugin


• 
  Unsloth — Fine-tuning tool with Dynamic 2.0 GGUF quantization


• 
  QMD — Local markdown search engine

Originally published at StarBlog