Skip to content

The Best Engine for a Local Agent on Qwen3.8-27B

Run MTPLX with Qwen3.8-27B-MTPLX-Optimized-Speed and --mtp --depth 3.

We measured eleven configurations across eight engines on one M5 Max. For agent work it wins on every metric that matters, and the second place is not close:

MTPLX Optimized-Speed llama.cpp b10434 +MTP Ollama 0.32.13
First token 99 ms 125 ms 240 ms
First token, 56k cold 301 ms 365 ms 574 ms
Prefix reuse 7,530 / 7,530 7,526 / 7,530 0 / 7,530
Throughput 44.4 tok/s 31.8 32.8
Memory 27 GB 36.8 GB 30.3 GB

That is +40% throughput, 26 ms less latency per turn, and 10 GB less memory than the best of the rest. We run it in production.

Why those metrics and not tokens per second

An agent is not a chat. It does not stream one long answer — it takes dozens of short turns, and every turn re-reads everything that came before. So the number you feel is time to first token, paid once per turn, and it depends almost entirely on whether the engine kept the previous prompt in cache.

The spread on that metric is 335× across our table — 99 ms to 33 seconds. The spread on throughput is 5%. That is the whole argument: for agent work, pick on latency and prefix reuse; throughput is a tiebreaker.

Two engines reuse at token level (MTPLX, llama.cpp). One reuses in 1,024-token blocks and re-prefills 1,386 tokens every single turn, forever (oMLX, 1,968 ms). Three reuse nothing at all and pay the full prompt each time — mlx-vlm at 9,982 ms, rapid-mlx at 33,195 ms. On a 60-turn agent session, that last one costs 33 minutes of pure waiting.

The full table

Warm and 56k are tok/s. First token on the warm turn; @56k on a cold 55,839-token prompt.

Engine Weights Warm At 56k First token @56k Memory tok/s/W
MTPLX Bare-Speed MTPLX 4-bit g64 56.0 46.5 101 ms 322 ms 22.0 GB 0.694
MTPLX Optimized-Speed MTPLX 4-bit g32 44.4 37.5 99 ms 301 ms 27.0 GB 0.623
mlx-vlm + MTP drafter MLX 4-bit ⁽¹⁾ 43.3 28.9 9,982 ms 100,733 ms 15.5 GB 0.744
MTPLX Optimized-Quality MTPLX 8-bit g64 40.8 30.3 118 ms 100,243 ms 32.9 GB 0.575
Ollama 0.32.13 GGUF (undeclared) 32.8 21.7 240 ms 574 ms 30.3 GB 0.451
llama.cpp b10434 + MTP GGUF Q5_K_XL ⁽²⁾ 31.8 22.7 125 ms 365 ms 36.8 GB 0.395
rapid-mlx 0.12.11 MLX 4-bit ⁽¹⁾ 30.2 24.7 33,195 ms 287,548 ms 15.0 GB 0.544
oMLX 0.6.0-dev oQ4e-mtp (third-party) 29.8 24.6 1,968 ms 1,991 ms 16.7 GB 0.512
mlx-lm 0.31.3 MLX 4-bit ⁽¹⁾ 28.8 24.0 432 ms 799 ms 14.6 GB 0.472
LM Studio 0.4.21 +MTP GGUF Q5_K_XL ⁽²⁾ 27.8 23.4 419 ms 825 ms 36.3 GB 0.422
LM Studio 0.4.21 defaults GGUF Q5_K_XL ⁽²⁾ 23.1 18.7 359 ms 865 ms 35.2 GB 0.351

⁽¹⁾ ⁽²⁾ = byte-identical weights file. MTPLX rows ran on 2.6.0. One cell, one export. All columns are medians of the warm phase. tok/s/W divides decode throughput by whole-SoC power (58 to 83 W depending on the engine) — not GPU-only, which ranks differently.

How to read it. Throughput gaps under 8% are noise — Ollama through LM Studio+MTP (32.8 to 27.8) are tied, not ranked. The two 100-second entries are cache misses, not engine speed: Optimized-Quality does 118 ms warm and returns to 498 ms on the repeated 56k turn. Memory is not comparable across families (llama.cpp mmaps: 36.8 GB resident is 19.4 GB physical).

The one thing that will surprise you: the server barely matters

mlx-vlm, mlx-lm and rapid-mlx served the same file, lmstudio-community/Qwen3.8-27B-MLX-4bit, snapshot 6067b15c, byte for byte. Two of them ran bare, without speculation:

30.2 against 28.8 tok/s. A 4.9% difference — below our own rerun noise.

Swapping the MLX server buys nothing. The third one, mlx-vlm, reached 43.3 — but it ran with the MTP drafter loaded. That +50% is the flag, not the server — see the appendix on multi-token prediction, which reaches the same conclusion from the other direction.

Choosing against a different constraint

Our recommendation optimises for agent work. If yours differs, the table already answers:

  • Smallest footprint → mlx-lm, 14.6 GB. You lose the prefix cache granularity.
  • Best battery life → mlx-vlm, 0.744 tok/s/W. Unusable for agents (9,982 ms first token), excellent for batch.
  • Raw throughput → MTPLX Bare-Speed, 56.0 tok/s. We advise against it: the quantization author publishes a divergence-from-bf16 of 0.0376 against 0.0220 for Optimized-Speed — 36× further from bf16 than the 8-bit build. His measurement, not ours, not reproduced.
  • You already run llama.cpp → stay. Add --spec-type draft-mtp and you close most of the gap; 125 ms first token is fine for an agent.

Run it with reasoning on

Qwen recommends xhigh for agentic work and warns that a lower effort "can lead to insufficient analysis, more failures, and repeated retries". MTPLX 2.7.1 separately lists disabled reasoning as a known issue for Qwen3.8. high does not exist on this model — low, medium, xhigh only, and xhigh is the default.

⚠️ Note the effort level is a chat-template variable, not an API field: sent as a normal request parameter it is silently ignored. It must be set at launch.

Four limits that change how you read this

  1. Nothing here ran a task to completion. Output was capped at 400 tokens and every run hit the cap. These are streaming throughput numbers on truncated continuations.
  2. Reasoning was off for all measurements, to make engines comparable. That is not how we deploy — see above. We have no figure on what reasoning costs each engine.
  3. Tuning was not symmetric. llama.cpp got explicit cache flags the MLX engines did not; MTPLX ran --profile turbo; sampling parameters and thermal limits differ per engine. Gaps between families are indicative, not clean.
  4. We are recommending the engine that alone reads its own quantization format, measured with our own tool, published on our own site. The MTPLX rows are the only ones where engine and weight format cannot be separated. Weigh it accordingly — the flag table in the appendix is the part of this page that costs us nothing to be right about.

Two engines with native MTP, vmlx and vllm-mlx, could not be measured in time.

Appendix: the MTP flag, engine by engine

Qwen3.8 ships a multi-token prediction head inside the weights. The model proposes several tokens ahead, the engine verifies them in one pass. Four of the six engines that support it ship it off.

Engine Flag On by default?
MTPLX ⁽ᵐ⁾ --mtp --depth 3 yes
vmlx ⁽ᵈ⁾ --native-mtp-depth yes
llama.cpp ⁽ᵐ⁾ --spec-type draft-mtp --spec-draft-n-max 4 no
LM Studio ⁽ᵐ⁾ --speculative-draft-mtp no
mlx-vlm ⁽ᵐ⁾ --draft-kind mtp --draft-model mlx-community/Qwen3.8-27B-MTP-4bit no
vllm-mlx ⁽ᵈ⁾ --enable-mtp no
Ollama · mlx-lm · rapid-mlx no support —

⁽ᵐ⁾ attested by our own launch commands and logs. ⁽ᵈ⁾ from project documentation only.

We measured what it costs to not know: same weights file, same 65,536 context, same thermal sequence, everything identical but two flags. LM Studio goes 23.1 → 27.8 tok/s, +19.9%. No GUI surfaces this. On mlx-vlm the same head is worth +50%.

Note the drafter is bundled in the GGUF for llama.cpp and LM Studio, but mlx-vlm needs a second repository downloaded alongside.

Check it is actually engaged — a flag accepted is not a flag working:

# llama.cpp / LM Studio — the log must mention a draft context at startup
grep -i "draft" server.log        # "creating MTP draft context"
# any OpenAI-compatible engine — acceptance rate should sit at 0.9+
curl -s localhost:8080/v1/chat/completions -d '…' | jq '.timings'

Reproducing this

Every number comes from a certified card produced by asiai, through a single scripted path with solitude gates, served-model identity proofs and thermal sampling. Raw exports, full launch commands and prompt text: ask and we publish them.

If you take one thing: check whether your engine has multi-token prediction, and whether it is on.