Running Qwen3 Through the ExecuTorch MLX Delegate: Up to 4.52x Faster on M1 Max

Hello, everyone.

There are now many ways to run an LLM on a Mac, but exporting a PyTorch model for Apple Silicon and executing it in a lightweight runtime is still an evolving path. How much faster is it, and does 4-bit quantization change the output?

Today, I am looking at ExecuTorch's experimental MLX delegate, released in May 2026. It enables PyTorch models to run on Apple Silicon GPUs. I use ExecuTorch 1.3.1 to run Qwen3-0.6B and compare it with PyTorch MPS.

The short result is that decode throughput was 41.8 tokens/s with PyTorch MPS BF16, 134.8 tokens/s with MLX BF16, and 188.9 tokens/s with MLX INT4. MLX INT4 was 4.52x faster, and its file was 71.8% smaller than BF16. However, INT4 changed the generated output in two of three simple prompts.

What Is the ExecuTorch MLX Delegate?

ExecuTorch is a runtime for running trained PyTorch models on desktops, phones, and embedded devices. Inference means feeding input into a trained model to obtain an output.

The is an LLM family announced by the Qwen Team on April 29, 2025. A single model can switch between a thinking mode for step-by-step reasoning and a non-thinking mode for shorter answers. The family supports more than 100 languages and dialects.

The


ExecuTorch



PyTorch
.

Reproducing the Lab

You need an Apple Silicon Mac, Xcode Command Line Tools, mise, uv, and an internet connection. The first run downloads the Qwen3 weights and creates about 1.53 GB of PTE files in total.

git clone --depth 1 --filter=blob:none --sparse \

  https://github.com/kiarina/labs.git

cd labs

git sparse-checkout set .gitignore .mise/tasks Makefile mise.toml \

  2026/07/22/executorch-mlx-qwen3

mise -C 2026/07/22/executorch-mlx-qwen3 run

The export and benchmark steps can also be run separately.

mise -C 2026/07/22/executorch-mlx-qwen3 run export

mise -C 2026/07/22/executorch-mlx-qwen3 run benchmark

Test Conditions

machine: MacBook Pro (Apple M1 Max, 32 GPU cores, 64 GB)

OS: macOS 26.5.2

Python: 3.13.7

ExecuTorch: 1.3.1

PyTorch: 2.12.1

Transformers: 4.56.1

model: Qwen/Qwen3-0.6B

generation: greedy, batch 1, up to 16 tokens

PTE: custom MLX SDPA / KV cache, requested maximum sequence 128

For performance, each backend was given the same Japanese prompt and forced to generate 16 tokens. The reported values are medians from five trials after warm-up. Prefill reads the input and produces the first token; decode generates the remaining tokens one at a time.

Each backend ran in a separate process. The MLX path loaded a fresh forward method and initialized its KV cache for each trial, while the PyTorch path created a fresh cache for every trial.

Results

PTE Size

INT4 was 856,601,220 bytes, or 71.8% smaller, than BF16. Quantization itself takes work, so exporting INT4 took about 13 seconds longer.

Generation Speed

The input was the Japanese prompt “Briefly explain local inference on Apple Silicon.”

MLX BF16 decoded 3.22x as fast as PyTorch MPS BF16, while MLX INT4 was 4.52x as fast. INT4 was also 1.40x faster than MLX BF16 and reduced the RSS increase by about 63%.

There are important qualifications. The MLX load figure measures only opening the PTE program, not all work needed to materialize weights for GPU use. RSS measures the process's main memory, not GPU memory itself. PyTorch reported 1.20 GiB of MPS driver-allocated memory at the end of the run. Because GPU memory was not measured on the same basis, the table does not prove that PyTorch used the least memory.

On the first invocation only, BF16 prefill took 0.434 seconds and INT4 took 1.014 seconds. Cold starts that include Metal setup and initial compilation were much slower than the warmed-up figures.

Generated Output

I compared the generated token sequences on three short prompts.

MLX BF16 matched PyTorch MPS BF16 token for token in all three cases. Within this narrow test, changing the execution path to MLX did not introduce a difference.

INT4 matched in only one case. Quantization represents numbers more coarsely, so when candidate next tokens have similar scores, their ranking can change.

The capital answer was wrong in both BF16 and INT4. The small 0.6B model, prompt, and greedy decoding could all contribute, but three questions are not enough to identify the cause. This probe checks differences between backends; it does not certify the model's knowledge or answer quality.

A Plain-Language Reading of the Results

1. The same small LLM ran substantially faster through MLX.

Even BF16 decoded 3.22x as fast as PyTorch MPS. Exporting a PyTorch model to a lightweight Mac runtime looks promising.

1. Four-bit weights are smaller and faster, but answers can change.

INT4 reduced the file from about 1.19 GB to 336 MB and produced the highest decode rate. Yet it changed two of only three token sequences. It should be evaluated on a task-specific quality set before adoption.

1. The gap cannot be attributed to MLX kernels alone.

This comparison covers an ExecuTorch MLX pipeline versus a Transformers/PyTorch MPS pipeline. Their runtimes, cache handling, and execution paths differ, so the numbers describe the complete pipelines.

Reproducibility Findings

The dependency metadata for executorch==1.3.1 allowed PyTorch 2.13.0, but importing the published ExecuTorch extension failed because the materialize_cow_storage symbol was missing. Pinning PyTorch to 2.12.1 made the same wheel work. The failure can be reproduced with this optional task:

mise -C 2026/07/22/executorch-mlx-qwen3 run probe-torch-2-13

The bundled PTE inspector also failed because its included flatc did not recognize the --json option. I verified full-graph delegation from the partitioner log emitted during export instead.

Limitations

• Only one M1 Max, Qwen3-0.6B, batch 1, and short Japanese prompts were tested.


• Five short trials do not control thermal state, power consumption, or other GPU workloads.


• Only BF16 and INT4 were compared; FP16, 2/8-bit, Core ML, and other paths were not tested.


• The quality probe had only three questions; no standard benchmark or perplexity was measured.


• The requested maximum sequence was 128, and long text was not tested.


• GPU memory could not be measured on the same basis for MLX and MPS.


• Cold-start latency was observed only once.


• The MLX delegate is experimental.

My Takeaway

Lowering the entire Qwen3-0.6B graph to MLX and increasing decode throughput by more than 3x without changing the BF16 tokens was a good result. INT4 reduced the file to less than one-third of its BF16 size and reached about 189 tokens/s, which is attractive when embedding a small model on a Mac.

The output changes from 4-bit quantization appeared immediately, even in this tiny probe. Quantization is not a free speedup. Paired with a quality suite for the real task, this path could work well for local helper features or small, responsive agents.