05 · Quantization Kernels in C

Quantization is the single reason a 70B model runs on a MacBook. It replaces each FP16 weight (2 bytes) with an INT4 or INT8 value plus a shared scale, cutting model size 3–4× and — because LLM decode is memory-bandwidth bound (see 03_cache_and_memory_hierarchy.md) — speeding up inference by nearly the same factor. This is a C-level game, not a Python-level one: the block formats, the packed nibble layouts, and the SIMD dot products all live in ggml-quants.c. Learn them once and you can read half of llama.cpp.

The math in one paragraph

Symmetric per-block quantization: pick a block of B fp32 values x[0..B-1]. Compute d = max(|x|) / q_max where q_max = 7 for INT4 or 127 for INT8. Store q[i] = round(x[i] / d) as an integer, and store d as an FP16 scale next to the block. To dequantize: x[i] ≈ q[i] * d. Error is bounded by d/2 per element. Asymmetric (Q4_1-style) adds a zero-point m so q[i] = round((x[i] - m) / d), giving one more parameter per block and slightly better accuracy on skewed distributions.

That’s it. The math is trivial. The engineering is where it lives.

The ggml block-format family (as of llama.cpp 2026)

Every format is a packed struct in ggml-common.h. Read that file top to bottom; it’s ~600 lines and worth every one.

Format	Block size	Per-block metadata	Bytes/block	Bits per weight	Use
Q4_0	32 elements	1× FP16 scale (d)	2 + 16 = 18	4.5	classic, fast
Q4_1	32	FP16 d + FP16 m (min)	2+2+16 = 20	5.0	asymmetric
Q5_0	32	FP16 d + 4 bytes hi-bits	2+4+16 = 22	5.5	—
Q5_1	32	FP16 d + FP16 m + 4 hi-bits	24	6.0	—
Q8_0	32	FP16 d	2 + 32 = 34	8.5	activation quant
Q8_1	32	FP16 d + FP16 sum	4 + 32 = 36	9.0	pair with Q4_1 dot
Q2_K	super-block of 256	hierarchical 4-bit scales + FP16 super-scale	~84	2.625	extreme compression
Q3_K	256 super	similar	~110	3.4375	—
Q4_K	256 super	8× sub-block scales in 6 bits + FP16	~144	4.5	the default recommended quant
Q5_K	256 super	similar	~176	5.5	—
Q6_K	256 super	8-bit scales + FP16	~210	6.5625	near-lossless
IQ2_XS, IQ3_XXS, IQ4_XS, …	256 super	codebook + imatrix	varies	2.0–4.5	codebook-based, needs importance matrix

Read the block_q4_0 and block_q8_0 struct definitions in ggml-common.h — they’re what appear in the working-copy of ggml as of mid-2026. The exact line numbers shift release to release; grep for typedef struct in that file.

The Q4_0 · Q8_0 dot product — the single most important kernel

When llama.cpp does matmul between a Q4_0 weight tensor and an activation vector, it first quantizes the activation to Q8_0 on the fly (fast, per-block), then dot-products two blocks of 32 elements. The kernel ggml_vec_dot_q4_0_q8_0 in ggml/src/ggml-cpu/ggml-cpu-quants.c is the hottest inner loop in the entire binary. Its shape:

1. Load 16 bytes of packed Q4_0 nibbles → unpack to two 32-lane int8 vectors (low nibbles and high nibbles, sign-extended to [-8, +7]).

2. Load 32 int8 activations from the Q8_0 block.

3. Do a 32-lane int8×int8 → int16 multiply and pairwise add. On AVX2: _mm256_maddubs_epi16 + a bias correction because Intel’s maddubs treats one operand as unsigned. On AVX-VNNI: single _mm256_dpbusd_epi32 instruction (a 2–4× improvement, introduced in Alder Lake/Sapphire Rapids). On NEON: vdotq_s32 (ARMv8.2-A DotProd) or vmlal_s8 fallback.

4. Multiply the accumulated int32 by d_a * d_b (both FP16 scales) and add to the FP32 accumulator.

This kernel is your Rosetta Stone. Once you can read it, Q4_K, Q5_K, Q6_K all follow the same pattern with extra sub-block bookkeeping.

Perplexity tolerances (from tests/test-quantize-fns.cpp)

Quantization is only useful if the model still works. llama.cpp enforces per-format RMSE tolerances in its test suite:

Format	Max RMSE per element	Notes
Q4_0, Q5_0, Q8_0	≤ 0.002	classic quants
Q4_1, Q5_1	≤ 0.0025	—
Q2_K	≤ 0.0075	extreme compression, larger error
IQ3_XXS	≤ 0.0050	codebook
NVFP4	≤ 0.0030	Nvidia 4-bit float format

On actual model quality: Q4_K_M adds only ~+0.1754 to perplexity on Llama-3-8B (from the QUANT_OPTIONS table in tools/quantize/quantize.cpp:34-74). For most workloads that is an invisible quality loss for a 3.3× memory reduction. This is why Q4_K_M is the default recommendation in the llama.cpp README.

The bandwidth argument, made concrete

On a Ryzen 7950X (~80 GB/s DRAM):

Format	7B model size	Tokens/sec ceiling (bandwidth ÷ size)
FP16	~14 GB	~5.7 tok/s
Q8_0	~7 GB	~11.4 tok/s
Q4_K_M	~4.1 GB	~19.5 tok/s
Q2_K	~2.7 GB	~29.6 tok/s

This is why quantization gives near-linear speedup. Every kernel writer at llama.cpp has this table in their head.

Practical llama.cpp benchmark (RTX 3090, Qwen3.5-35B Q4_K_M, mid-2026)

Runtime	Config	Tokens/sec
llama.cpp	--cache-type-k q8_0 --cache-type-v q8_0 -np 1	~100
llama.cpp	defaults	~50
Ollama (wraps ggml)	defaults	~15–20

The 5× gap between hand-tuned llama.cpp and Ollama is entirely KV cache quantization + explicit batching flags. Nothing exotic. This is what “knowing your tool” is worth in wall-clock.

What most people get wrong about quantization

They think quantization is a model concern — something you do once at export time and forget. It isn’t. Quantization is a runtime memory layout concern. Every choice (block size 32 vs 256, symmetric vs asymmetric, scale in FP16 vs FP32, K-super-blocks vs flat) is a trade-off between (a) SIMD friendliness, (b) DRAM bandwidth used, and (c) accuracy. You cannot pick a format without also picking a dot-product kernel that consumes it. This is the reason ggml has 20+ formats — each is a different point on that trade curve for a different chip, a different bandwidth budget, a different accuracy tolerance. Learn to read the block struct + kernel as a pair, never separately.

---

Return to README.md · Next: 06_reading_llama_cpp.md