10 — GGUF k-quants and i-quants: The Bit-Level Reality

This is r/LocalLLaMA’s native tongue. When someone asks “why is IQ4_XS quality-competitive with Q5_K_S at half a bit less?”, you should have the answer at bit-level. This is also the file where the on-prem CRM angle becomes concrete: your Zoho customer on a Xeon+DDR5 box is running Q4_K_M whether they know the name or not.

The GGUF file format (30-second version)

GGUF (GPT-Generated Unified Format) is llama.cpp’s successor to GGML. A single-file, memory-mappable binary layout designed for fast partial loading on consumer hardware. Structure:

[Magic "GGUF"] [Version] [Tensor count] [Metadata KV count]
[Metadata KV pairs …]        ← model config, tokenizer, chat template, everything
[Tensor info array …]        ← name, dtype, shape, offset
[Padding to alignment]
[Tensor data blob]           ← mmap-able, dtype-mixed

Project 1 in 14_projects.md: parse the header of a real GGUF file in ~50 lines of Python using the struct module. It’s the fastest way to understand this format, and it puts you ahead of 99% of “I run local LLMs” hobbyists.

The metadata KV format supports arbitrary types (ints, floats, strings, arrays), which is how tokenizer merges, chat templates, and RoPE parameters all live in the file. This is why GGUF is self-describing — you can llama-cli --model foo.gguf with no configs.

The quantization type taxonomy

llama.cpp shipped many quant types over the years. The ones that matter in <phone_number_or_numberic_id_or_random_id_150>:

Legacy (Q4_0, Q4_1, Q5_0, Q5_1, Q8_0)

Block size 32, one fp16 scale (Q4_0/Q5_0/Q8_0 symmetric) or fp16 scale + fp16 min (Q4_1/Q5_1 asymmetric). Simple. Still used for Q8_0 (near-lossless, ~8.5 bits/weight, always the reference “safe” quant).

k-quants (Q3_K, Q4_K, Q5_K, Q6_K) with S/M/L variants

The main event since 2023. “K” stands for the super-block structure. All k-quants use:

  • Super-block: 256 weights = 8 sub-blocks × 32 weights each.

  • Sub-block scale storage: 6-bit per sub-block.

  • Super-block scale: fp16 min + fp16 delta.

  • Sub-block layout: 6-bit min + 4/5/6-bit weights depending on the K variant.

The Q4_K_M layout, bit by bit (memorize this)

One super-block of Q4_K_M = 144 bytes for 256 weights = 4.5 bits per weight.

Struct layout of one Q4_K super-block (256 weights):

┌──────────────────┬──────────────────────────────────────────────┐
│ 2 bytes          │ fp16 super-block delta (d)                   │
├──────────────────┼──────────────────────────────────────────────┤
│ 2 bytes          │ fp16 super-block min (dmin)                  │
├──────────────────┼──────────────────────────────────────────────┤
│ 12 bytes         │ 8 × 6-bit sub-block scales + 8 × 6-bit mins  │
│                  │ packed as (6-bit scale, 6-bit min) × 8       │
│                  │ = 96 bits scales + 96 bits mins = 192 bits   │
├──────────────────┼──────────────────────────────────────────────┤
│ 128 bytes        │ 256 × 4-bit quantized weights                │
│                  │ = 8 sub-blocks × 32 weights × 4 bits         │
└──────────────────┴──────────────────────────────────────────────┘
Total = 2 + 2 + 12 + 128 = 144 bytes for 256 weights = 4.5 bits/weight

Dequant math:

For sub-block s ∈ [0..7], weight i ∈ [0..31]:
  scale_s = d    · (sub_scale_s / 63)      # 6-bit unsigned, max 63
  min_s   = dmin · (sub_min_s   / 63)
  x = scale_s · q_{s,i} - min_s            # q_{s,i} is the 4-bit weight

That’s the whole thing. Two levels of scale (super-block gives dynamic range, sub-block gives fine-grained fit), asymmetric (min offset for non-symmetric weight distributions).

Q_K_S vs Q_K_M vs Q_K_L (the M/S/L suffixes)

These are not different sub-block schemes — they are the same k-quant type applied to different weight tensors in the model:

Suffix

Common tensors upgraded

Bit cost

_S (Small)

All tensors at the base quant

Lowest

_M (Medium)

attn_v.weight, ffn_down.weight upgraded to next-higher quant

Moderate

_L (Large)

Even more sensitive tensors upgraded

Higher

So Q4_K_M = Q4_K base, with attn_v and ffn_down at Q6_K instead. This is why Q4_K_M is the sweet spot — the tensors that need protection get more bits, and everything else stays cheap. The choice of which tensors to upgrade came from empirical PPL sweeps by Georgi Gerganov and contributors.

Bits/weight table for the popular ones (Llama-3-8B, approximate; varies slightly by architecture):

Quant

Effective bits/weight

Notes

Q2_K

~2.6

Aggressive; use only when memory-desperate

Q3_K_S

~3.4

Q3_K_M

~3.9

Q4_K_S

~4.6

Q4_K_M

~4.85

The default. Sweet spot.

Q5_K_S

~5.5

Q5_K_M

~5.7

Near-fp16 for most models

Q6_K

~6.6

Basically lossless

Q8_0

~8.5

Safe reference — always the “is my quant broken?” check

i-quants (IQ2_XXS, IQ3_XXS, IQ4_XS, IQ4_NL, IQ1_S, IQ1_M)

The 2024 evolution: importance-matrix guided, non-uniform codebook quantization. Two ideas fused:

  1. Non-uniform codebook. Instead of q ∈ {0..15} mapping linearly to weights, use a small lookup table of ~15 values that better matches the empirical weight distribution (bell-shaped, so more codepoints near zero). NF4-style, but tiny per-block LUTs.

  2. Importance matrix (imatrix). Per-weight sensitivity computed from calibration activations — essentially, GPTQ-style Hessian diagonal without the full Hessian. Used to bias rounding toward preserving high-impact weights.

Generate the imatrix once:

./llama-imatrix -m llama-3.1-8b-F16.gguf \
                -f calibration.txt \
                -o llama-3.1-8b.imatrix \
                --chunks 100

Then use it when quantizing:

./llama-quantize --imatrix llama-3.1-8b.imatrix \
                 llama-3.1-8b-F16.gguf \
                 llama-3.1-8b-IQ4_XS.gguf \
                 IQ4_XS

Why i-quants beat k-quants at the same bit budget below Q4:

  • Q3_K_M is ~3.9 bits/weight, PPL loss on Llama-3-8B ~0.15.

  • IQ3_XXS is ~3.06 bits/weight, PPL loss on Llama-3-8B ~0.20.

  • IQ4_XS is ~4.25 bits/weight (0.6 bits less than Q4_K_M!), PPL loss ~0.05.

Below ~4 bits, non-uniform + imatrix is a real accuracy jump. This is why IQ4_XS is now the default “aggressive but still good” quant on r/LocalLLaMA for models where you want to squeeze extra context in.

Compute cost: i-quants are slower to quantize (more work per weight) and slightly slower to dequantize (LUT lookup vs pure arithmetic). On CPU the LUT can hurt; on GPU it’s fine because you’re memory-bound anyway.

The CPU quant kernels (the un-glamorous heroes)

llama.cpp’s real magic is not the format — it’s the hand-tuned CPU dequant+matmul kernels. In ggml/src/ggml-cpu/:

  • AVX2 / AVX-512 paths for x86: SIMD dequant into f32 registers, then vfmadd into f32 accumulators, then quantize the input activations on the fly (Q8_K “working type”).

  • AVX-512 VNNI (Intel Xeon Sapphire Rapids+) — int8 dot products in one instruction, ~2–4× speedup over pure AVX-512.

  • AMX (Advanced Matrix Extensions, Sapphire Rapids+ / Granite Rapids) — int8/bf16 tile matmul; llama.cpp added AMX paths in <phone_number_or_numberic_id_or_random_id_151>. This is what makes DDR5 Xeon boxes competitive with 3090s for 70B-Q4 chat throughput.

  • NEON / SVE paths for ARM (M1/M2/M3, Ampere Altra, GB200 Grace CPUs).

  • Metal shaders for Apple silicon (MPS/GPU path, distinct from NEON CPU path).

The core trick is always the same: load a k-quant block from memory, dequantize into 32 fp16 values in registers, multiply by an int8-quantized activation block, accumulate in fp32. Fused into one loop. Same idea as Marlin, just on CPU/SIMD instead of GPU/tensor-core.

Zoho translation: for on-prem CRM inference on hardware that doesn’t have a GPU (SMB customers, air-gapped legal/healthcare), a Sapphire Rapids Xeon with 8-channel DDR5-4800 and AMX gives you ~350 GB/s bandwidth. That runs Llama-3-8B-Q4_K_M at ~15–25 tok/s decode, one user, no GPU. Knowing this number lets you sell a real product on a Xeon that the customer already owns.

The imatrix flow in more detail

The importance matrix is essentially a diagonal Hessian estimate, computed by hooking activations during a calibration forward pass. Per-weight, per-tensor sensitivity is stored as f32 values in a small companion file (~KB to MB per model). The imatrix biases rounding at quantization time:

naive:      q_i = round(w_i / s)
imatrix:    q_i = round(w_i / s)  +  bias derived from ∂loss/∂w_i squared

The intuition mirrors GPTQ’s Hessian intuition (04_gptq.md): a weight with high derivative squared matters more, so it should get lower rounding error. i-quants use the imatrix in two places: to guide which codebook entries are used, and to reorder / re-map within each block.

Calibration data matters. Use domain-representative text. Standard practice: 100-500 chunks of ~512 tokens from a mix of WikiText, GitHub code, and if possible your target domain (chat traces, CRM ticket text). Undersize calibration → i-quants can underperform k-quants at the same bits.

Where GGUF k-quants win, where they lose

Win:

  • CPU + hybrid inference — no other ecosystem is close on CPU.

  • Consumer hardware — mmap + partial GPU offload (-ngl N) lets you run 70B on 24GB VRAM + 64GB DDR5.

  • Small model quality — the imatrix flow lets 1-3B models survive Q4/Q3 better than pure GPTQ.

  • Portability — one file, ships everywhere, self-describing.

Lose:

  • High-batch GPU serving — llama.cpp’s server is not a scheduler. For batch >4 on GPU, vLLM/SGLang with GPTQ/AWQ/Marlin destroys it.

  • Modern PTQ techniques — no rotation-family methods (QuaRot/SpinQuant/DuQuant) in GGUF-land. i-quants use imatrix but not orthogonal transforms.

  • Non-uniform hardware — Blackwell FP4/MXFP4 native paths are just now landing in llama.cpp. Behind the curve on cutting-edge hardware.

The GGUF community discipline

Read a few real quantization posts to learn the vibe:

  • The bartowski and mradermacher HuggingFace repos are the reference imatrix-quantized model distributions. They ship Q4_K_M, Q5_K_M, IQ4_XS, IQ3_M, etc. for every popular model within hours of release.

  • Their PPL tables (usually WikiText-2, sometimes English-only slices) are the community’s quality reference.

  • The naming convention model-name-{quant}.gguf is standard.

Contribution ladder: run the PPL sweep yourself for a model that hasn’t been done, publish, get called useful in the r/LocalLLaMA thread.

Anti-patterns

  • Don’t confuse GGUF’s Q4 with GPTQ’s W4. Different bit layouts, different granularities, different kernels. A Q4_K_M file will not load in vLLM. A GPTQ-quantized safetensors will not load in llama.cpp.

  • Don’t skip imatrix for IQ_ quants. Vanilla IQ3_XXS without an imatrix underperforms Q3_K_M. The imatrix is not optional.

  • Don’t Q4_K_M every layer of a small model. For <3B models, i-quants (IQ4_XS) or straight Q5_K_M usually beat Q4_K_M in quality per bit.

  • Don’t benchmark llama.cpp with -t 32 on a 16-thread CPU. Oversubscribing threads is the #1 reason for surprising slowdown reports on Reddit.

The two-sentence study answer

GGUF Q4_K_M packs 256 weights into a 144-byte super-block: fp16 super-delta + fp16 super-min + 12 bytes of 8×6-bit sub-scales/mins + 128 bytes of 4-bit quantized weights, giving 4.5 bits/weight with the _M variant additionally upgrading attn_v and ffn_down to Q6_K — that’s the entire recipe behind why Q4_K_M is the r/LocalLLaMA default. Below ~4 bits, non-uniform codebooks + importance-matrix guided rounding (i-quants like IQ4_XS) beat k-quants at the same bit budget, which is why bartowski’s quant repos ship both.

Homework

  1. Write a Python script (~50 lines) that parses a GGUF header and lists every tensor name, shape, dtype, and offset. Use struct only, no llama.cpp bindings.

  2. Compute by hand: how many bytes does a Llama-3-8B model take in Q4_K_M? (8B params × 4.5 bits / 8 = ~4.5 GB. Verify against the actual .gguf file size on HF.)

  3. Quantize Llama-3.1-8B to Q4_K_M and IQ4_XS with imatrix, using the same 100-chunk calibration. Report file sizes, WT2 PPL, and CPU decode tokens/sec on your machine.

  4. Read the llama.cpp/ggml/src/ggml-quants.c file: pick one function, e.g. quantize_row_q4_K_ref, and annotate every line. This is the reference implementation the SIMD paths mirror.