I tried four LLM speedups on CPU. Three made it slower. One worked but vetoed itself.

There's a comfortable assumption in the small-model-on-CPU world: if stock llama.cpp is your baseline, every clever trick is a free win. Speculative decoding gets you "2-3× faster." ik_llama.cpp's IQK matmul kernels are "the optimized fork." TurboQuant claims "8× faster" on the abstract. The decision feels obvious — try the trick, take the win, move on.

I measured all four against stock llama.cpp on a single shared Xeon E-2176G box (4 cores, cgroup-isolated, no GPU). 35 BFCL tool-calling cases per cell, ~4B parameter Q4_K_M models (Qwen 3.5 4B, Gemma-4-E4B-it, Phi-4-mini-instruct). Eleven cells of data. The headline:

Stock llama.cpp + Gemma-4-E4B-it wins. No fork. No quantized KV. No draft model. 94.3% tool-calling accuracy, 6.2s p50, 100% on parallel calls. Every "speedup" I tried made the actual workload worse.

The story is which speedup broke in which way. They all broke for different reasons. The boring answer wins on credibility.

1. Speculative decoding — 1.48× slower

Speculative decoding pairs a small "draft" model with a larger "target" model. The draft proposes tokens; the target verifies. Published numbers say ~2.5-3× speedup on CPU. Those numbers are honest — but they're measured on 7B+ targets.

On a 4B target with a 1.5B draft on a 4-core cgroup, the orchestration cost — running two models, swapping their KV caches, the rejection logic on every verification step — costs more than the drafts save. End-to-end the cell ran at 1.48× of baseline latency, in the wrong direction.

# what I ran
llama-server \
  --model Qwen3.5-4B-Q4_K_M.gguf \
  --model-draft Qwen3.5-1.5B-Q4_K_M.gguf \
  --draft-max 16 --draft-min 4 \
  --jinja --ctx-size 4096

The lesson is the workload, not the technique. If you have a 7B+ target and a 1.5B draft, the math probably still favors specdec. At 4B, it doesn't. Publishing-grade benchmark numbers are valid in their narrow regime; the regime where you actually ship is often a different one.

2. ik_llama.cpp — 1.53× faster end-to-end on Qwen, but parallel calls collapse 80% → 0%

ik_llama.cpp (ikawrakow/ik_llama.cpp) ships heavily-optimized IQK matmul kernels — generally accepted as the fastest llama.cpp fork on CPU. I measured 1.53× faster end-to-end on Qwen 3.5 4B (35 cases). That's a real win.

Until I ran the parallel-tool-calling subset.

Stock llama.cpp + Qwen → 80% on parallel calls. ik_llama.cpp + Qwen → 0% on parallel calls.

The parallel surface is structurally different — multiple tool calls in one response, returned as a JSON array. Something in ik_llama's sampling path drops them. Same model weights. Same chat template. Same prompts. Different fork.

On Gemma-4-E4B, the speedup shrank to 1.05× (Gemma was already efficient — it's the model the matmul kernels least helped) and parallel calls still broke. On Phi-4-mini, the fork wouldn't even load — Phi's tokenizer config hit an assertion at startup.

The verdict: a 1.53× single-call speedup that breaks parallel calls is a hard veto for production. Tool-calling agents don't ship if their parallel surface drops to zero.

3. TurboQuant — works, but trades the wrong axis at 4K context

I had to revert a wrong-answer commit on this one — full story in another article. The short version:

• TurboQuant does have a CPU AVX2 path. PR #21089 by elusznik in ggml-org/llama.cpp.
• Two cache types: tbq3_0 (5.19× KV compression) and tbq4_0 (3.94×).
• On real CPU end-to-end: ~2.2× slower generation, 17 pp accuracy drop on Qwen.

# the runnable check, once PR #21089 is built
llama-server --model <model.gguf> \
  --cache-type-k tbq3_0 \
  --cache-type-v tbq3_0 \
  --jinja --ctx-size 4096

TurboQuant is trading the wrong axis for most workloads. It compresses KV cache memory. That matters when context is 16K+. At 4K — most tool-calling — you don't need the savings, and you pay the throughput cost anyway. Memory savings real; speed wins conditional on a context window most tool-calling workloads don't have.

4. Phi-4-mini's default — 0.0% pass rate

Not a speedup, but worth ending on. Microsoft's Phi-4-mini-instruct on --jinja tool-calling: 0.0% across 35 cases. The chat-template parser falls through; the OpenAI-compatible client sees empty tool_calls every time. A 30-character system-prompt workaround rescues 74.3%, but parallel calls degrade.

If you're choosing a model on benchmark cards alone, Phi-4-mini looks competitive. In a real integration, it ships broken. Gemma-4-E4B doesn't.

The boring answer that won

Stock ghcr.io/ggml-org/llama.cpp:full + gemma-4-E4B-it at Q4_K_M + FP16 KV cache + --jinja --reasoning off --reasoning-budget 0. That's the ship recipe. It beats Qwen 3.5 4B and Phi-4-mini-instruct outright on the same hardware. It survives the parallel-tool-calling subset. It works with vanilla OpenAI client libraries with no fork-specific patching.

# the entire ship config
docker run -d --cpus=4 --memory=12g \
  -v /models:/models:ro -p 8080:8080 \
  ghcr.io/ggml-org/llama.cpp:full \
  --model /models/gemma-4-E4B-it-Q4_K_M.gguf \
  --threads 4 --ctx-size 4096 \
  --jinja --reasoning off --reasoning-budget 0

The meta-lesson

Engineering decisions on "is X faster than Y?" do not survive contact with your actual workload. Published numbers tell you the technique works in some regime. Whether that regime is yours is a measurement problem, not an inference problem.

Three things that look like wins on the abstract didn't survive my workload. One thing that nobody describes as a "trick" — running the stock build — was the boring, complete, ship-ready answer.

If the only thing you take from this: before you reach for a clever fork, measure your own workload against the stock baseline first. Then measure parallel calls. Then measure end-to-end, not single-token. Then decide.

Where the rest lives

• Full engine bake-off article (with the rest of the numbers and the cell-by-cell breakdown): deemwar-products.github.io/llama-cpu-benchmarks/article-engines
• The TurboQuant correction (separate piece, the one I had to revert): /article
• Raw cells (JSON, MIT, reproducible): deemwar-products/llama-cpu-benchmarks
• More like this at deemwar.com

— deemwar

References

• ggml-org — llama.cpp: LLM inference in C/C++ (2024). ggml-org/llama.cpp on GitHub
• Iwan Kawrakow — ik_llama.cpp: llama.cpp fork with IQK matmul kernels (2024). ikawrakow/ik_llama.cpp on GitHub
• Leviathan, Kalman, Matias — Fast Inference from Transformers via Speculative Decoding (2023). arXiv:2211.17192
• Berkeley Sky Lab (Gorilla team) — Berkeley Function-Calling Leaderboard (BFCL) (2024). gorilla.cs.berkeley.edu/leaderboard.html
• Google — Gemma 3n E4B Instruct model card (2025). huggingface.co/google/gemma-3n-E4B-it
• Deemwar — LLM CPU Benchmarks: reproducible cells and harness (2026). deemwar-products.github.io/llama-cpu-benchmarks