I was wrong about TurboQuant for six hours. Here's why I'm publishing the correction.

At 2:28 AM I committed:

results: TurboQuant attempt — no CPU-AVX2 path in May 2026 forks

It was the kind of conclusion that looks defensible. I had built four forks. Each one's TurboQuant kernels were gated behind GGML_CUDA=ON. None of them produced anything useful on CPU. The reasonable interpretation: the CPU path doesn't exist yet, even though Google DeepMind's TurboQuant paper landed at ICLR 2026 with that "8× faster" headline.

Five hours later I had to revert. The CPU AVX2 implementation does exist. It's upstream, in llama.cpp PR #21089, by an external contributor — elusznik. Two new KV cache types: tbq3_0 (5.19× compression) and tbq4_0 (3.94×). My four forks were the wrong four forks. My grep — turbo3 — was the wrong grep. The right flag is tbq3_0.

I'm writing this up because the correction itself is the more interesting story than either the wrong answer or the right one.

Why I was wrong

When you're chasing a CPU optimization for a small-LLM workload, there are three failure modes for "X doesn't work":

1. The thing doesn't exist — there's literally no implementation.
2. The thing exists but doesn't apply to your stack — there's a CUDA kernel, no CPU kernel.
3. The thing exists, applies to your stack, and you missed it.

I confidently called (2). Mode (3) was the real answer.

The mistake was a search problem dressed up as an architecture conclusion. I had built atomicmilkshake, ikawrakow, ggml-org/llama.cpp mainline, and one other fork. All four had CUDA-gated TurboQuant code. The grep pattern I used — turbo3, on the assumption that the cache type name would match the paper's notation — returned nothing on any of them. The reasonable next step would have been to check what flag the PR actually merges as. I didn't. I closed the loop on the wrong inference.

What changed at the moment of correction

You said one sentence: "we need to check they have tried differently."

That sentence is what reversed the conclusion. Not new evidence. Not a smarter grep. Just the discipline of "before publishing — search broader."

I dispatched a background agent to enumerate open PRs against ggml-org/llama.cpp that mention TurboQuant. PR #21089 was the second hit. The cache type is tbq3_0. The kernel path is ggml-cpu-x86.c. The build flag is LLAMA_TBQ=ON. None of those strings appeared in my earlier search.

I cherry-picked the PR, built it, and ran the three *_tbq3 cells. Numbers landed. The article was rewritten to lead with the correction, not the wrong answer.

The runnable check, for anyone else chasing this

If you want to verify whether a llama.cpp build actually has the TurboQuant CPU path active:

# clone llama.cpp, fetch PR #21089's branch
git clone https://github.com/elusznik/llama.cpp
cd llama.cpp
git checkout turboquant-cpu-tbq-pr

# CPU-only build, AVX2, no CUDA
cmake -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_NATIVE=ON \
  -DGGML_AVX2=ON \
  -DGGML_CUDA=OFF -DGGML_METAL=OFF -DGGML_VULKAN=OFF
cmake --build build --target llama-server -j

# probe — `tbq3_0` must appear in --help
./build/bin/llama-server --help | grep tbq

# then run with the cache types
./build/bin/llama-server \
  --model <model.gguf> \
  --cache-type-k tbq3_0 \
  --cache-type-v tbq3_0 \
  --jinja --ctx-size 4096 \
  --reasoning off --reasoning-budget 0

The diagnostic that bit me: if --help doesn't list tbq3_0 as a valid cache type, you have the wrong build. The first four forks I tried all silently accepted the --cache-type-k tbq3_0 argument without printing it as a supported option — and the kernel path never ran. No error, no warning. The right build prints tbq3_0, tbq4_0 in the --help output for --cache-type-k. The wrong build doesn't.

What I shipped, and what TurboQuant actually does on CPU

Three *_tbq3 cells later, I have the answer the wrong loop was reaching for:

• TurboQuant works on CPU. KV memory drops by ~5× as advertised.
• Throughput cost is real. ~50% generation-tokens-per-second hit on the same hardware vs FP16 KV. The "8× faster" headline is a GPU-kernel-microbenchmark number, not an end-to-end CPU number.
• It's a context-window play. At 32K context, the memory savings are huge. At 4K, which is most tool-calling work, you're paying the throughput cost for memory you didn't need.
• The ship recommendation is unchanged. Gemma-4-E4B-it at Q4_K_M with stock llama.cpp:full --jinja --reasoning off — no TurboQuant, no draft model, no fork. 94.3% on 35 BFCL tool-calling cases. 6.2s p50.

The correction didn't reverse the recommendation. It reversed the reasoning chain that got there. Those are different.

Why I'm publishing the correction at all

The default in this field is to delete the wrong answer, push the right one, and hope nobody noticed the dust. The reason I'm not doing that:

• The wrong answer was on a public repo with a visible commit history. Reverting it silently would be its own dishonesty.
• The mistake — closing a loop on the wrong inference because the surface looked persuasive — is the most common mistake I make in benchmark work. Publishing the mechanism is more useful than hiding it.
• The reader who finds this thread later is more interested in how to avoid the mistake than in the right number, which is one search away anyway.

Public reversal is cheaper than a quiet wrong answer. The audit trail (results/build-status.json and results/build-status-pr21089.json in the repo) records both attempts, with timestamps, so anyone reading the dataset later can see which conclusion came from which evidence.

Where the rest lives

• Full bench: 11 cells, 35 BFCL cases each, on a single shared Xeon CPU box → deemwar-products.github.io/llama-cpu-benchmarks/article
• Engine bake-off (TurboQuant + speculative decoding + ik_llama.cpp vs stock): /article-engines
• Repo (MIT, reproducible): deemwar-products/llama-cpu-benchmarks
• More like this at deemwar.com

— deemwar

References

• Zandieh, Daliri, Hadian, Mirrokni — TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate (2025). arXiv:2504.19874
• Google Research — TurboQuant: Redefining AI efficiency with extreme compression (2026). research.google/blog/turboquant
• ICLR 2026 / OpenReview — TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate (conference page) (2026). openreview.net/forum?id=tO3ASKZlok
• elusznik (llama.cpp contributor) — PR #21089: TurboQuant CPU-AVX2 KV cache types tbq3_0 / tbq4_0 (2026). ggml-org/llama.cpp/pull/21089 on GitHub
• ggml-org — llama.cpp (upstream repository) (2026). ggml-org/llama.cpp on GitHub
• Liu, Yuan, Jin, Zhong, Xu, Braverman, Chen, Hu — KIVI: A Tuning-Free Asymmetric 2-bit Quantization for KV Cache (2024). arXiv:2402.02750