01 — Quantization Theory: Affine Maps, Granularity, and the Dequant Math

Goal of this file: By the end, you can write down the affine quant equations by hand, draw the granularity ladder from memory, and derive the dequant cost of each granularity choice in bytes/element.

1. The one equation

Uniform affine quantization maps a real number x to a low-bit integer q:

q = round(x / s) + z            (quantize)
x̂ = s * (q - z)                 (dequantize)

Where:

  • s ∈ ℝ⁺ is the scale (fp16 or fp32).

  • z ∈ ℤ is the zero-point (integer, in the same range as q).

  • q ∈ {q_min, ..., q_max} — for INT4 signed: [-8, 7]; INT4 unsigned: [0, 15]; INT8 signed: [-128, 127].

Everything else in this phase is a variation of how many s and z you have and which values they cover.

The dual view: codebook

Equivalently, uniform affine quantization = a codebook of 2^N values equally spaced with step s, offset by z. “Non-uniform” methods (NF4, log-quant, k-means codebooks) drop this equal-spacing constraint. NF4 (07_fp8_family.md gets the intuition; 10_gguf_k_quants.md shows the block layout) uses codepoints spaced by quantiles of the standard normal — optimal for weight distributions that look approximately N(0,σ²).

2. Symmetric vs asymmetric

Asymmetric (has a nonzero z)

Covers an arbitrary interval [α, β] with a nonzero zero-point:

s = (β - α) / (q_max - q_min)
z = round(q_min - α / s)

Good for activations (post-ReLU, post-GELU) that are one-sided or heavily shifted. Cost: you carry a zero-point everywhere and pay for (q - z) in the dequant.

Symmetric (z = 0)

Covers [-α, α] (signed) or [0, α] (unsigned):

s = α / q_max     (with α = max(|x|))
z = 0

Good for weights — they’re roughly zero-mean and roughly symmetric after LayerNorm/RMSNorm. The dequant simplifies to a single multiply: x̂ = s·q. This is the reason almost all weight-only schemes are symmetric — the kernel epilogue is shorter.

Rule of thumb: weights → symmetric; activations → asymmetric (unless you’ve already migrated outliers via SmoothQuant/QuaRot, in which case symmetric activations become viable).

The min-max vs percentile debate

Both α = max(|x|) (min-max) and α = percentile(|x|, 99.9) are used. Min-max is exact but outliers destroy your resolution (one massive activation wastes 90% of your codebook). Percentile clips those outliers to ±α but preserves resolution for the bulk of the distribution. Nearly every modern method (GPTQ, AWQ, llm-compressor) exposes a clip_ratio / search_scale knob — that knob is choosing this tradeoff for you.

3. The granularity ladder

This is the ladder you climb from crude to sophisticated. Each rung buys accuracy by paying storage for more scales/zero-points.

per-tensor  →  per-channel  →  per-group  →  per-block-2D
(one s,z)      (one per row)   (one per       (one per
                                block of K)     tile of KxK)

Rung 1: per-tensor

One (s, z) for an entire weight matrix W ∈ ℝ^(M×N). Storage overhead: negligible. Accuracy: terrible for LLMs — a single outlier row can force s to be huge and destroy resolution for the other 4095 rows. Nobody uses per-tensor for weights in LLM PTQ, except in demos.

Rung 2: per-channel (a.k.a. per-row for weights, per-column for activations)

One (s, z) per output channel of W (i.e., one per row if you’re computing Wx, one per column if xW). For a Llama-3-8B q_proj of shape [4096, 4096], that’s 4096 scales.

  • Storage: 4096 * 2 bytes (fp16 scales) ≈ 8 KB — rounding error next to 4096²·0.5B = 8 MB of INT4 weights.

  • Accuracy: much better — each output channel has its own dynamic range.

This is the natural granularity for weights because activations flowing into that row see one scale; the whole matmul epilogue becomes a per-row scale-vector broadcast.

Rung 3: per-group (a.k.a. group-wise)

Split each channel into groups of size g (typically 128 or 64) along the reduction (input) axis, one (s, z) per group.

  • For W ∈ [4096, 4096] with g=128: 4096 * (4096/128) = 131,072 scales. In fp16 that’s 256 KB — still <4% overhead vs 8 MB of INT4.

  • Accuracy: this is the sweet spot. Almost all serious W4 methods (GPTQ, AWQ, GGUF Q4_K_M, NVFP4) live here.

Why 128? It’s the tile size CUDA kernels love. Marlin, Machete, and CUTLASS W4A16 kernels all bake g=128 into their tile geometry. g=64 gives ~0.5% better PPL and ~25% slower kernels — not worth it for most workloads.

Rung 4: per-block 2D (DeepSeek FP8 style)

One (s, z) per 2D tile of shape e.g. 128×128 or 1×128. This is essentially per-group extended to both dims. DeepSeek-V3 trained their weights in fp8 using per-tile (128, 128) block scaling for weights and (1, 128) per-token-group scaling for activations.

Rung 5: micro-blocks (MX/NVFP4 territory)

OCP MXFP4 uses group size 32 with a shared E8M0 (power-of-two-only) scale per group. NVFP4 uses group size 16 with an FP8-E4M3 scale per group plus a per-tensor FP32 scale. These small groups are the reason FP4 is remotely usable — you’re spending scale bytes to control outliers at 32× finer granularity than group-128.

Bit-level layouts covered in 08_fp4_mxfp4_nvfp4.md.

4. Storage math: “how much does the scale actually cost me?”

For an INT4 weight of shape [M, K]:

Granularity

Scale storage

Effective bits/weight

per-tensor

2 B (fp16)

4.000

per-channel

M * 2 B

4 + 2·16/K ≈ 4.008 (K=4096)

per-group g=128

M * K/g * 2 B

4 + 2·16/g = 4.25

per-block 2D 128×128

(M/128)·(K/128)·2 B

4 + 2·16/(128²) ≈ 4.002

MXFP4 (g=32, E8M0 scale)

M * K/32 * 1 B

4 + 8/32 = 4.25

NVFP4 (g=16, E4M3 + per-tensor FP32)

M * K/16 * 1 B + 4B

4 + 8/16 = 4.50

The true bits/weight is what matters when you’re comparing “true 4-bit” numbers on paper — NVFP4 is really 4.5 bits/weight. This is honest accounting; anyone who claims their 4-bit method is exactly 4.0 bits/weight is not counting scales.

5. Dequant math for kernels

The kernel-relevant question: for each fp16 multiply-accumulate, how many bits must I fetch from HBM, and how many instructions does dequant cost?

For W4A16 with group-128 symmetric quant, the inner loop looks like:

for each output tile:
  for each K-group of 128:
    load 128 nibbles of q_w              # HBM: 64 bytes
    load 1 scale s                        # HBM: 2 bytes
    load 128 fp16 x                       # HBM: 256 bytes (activations)
    for each of 128:
      w_fp = s * (q_w - z)                # register: 1 sub + 1 mul + zero-point handle
      acc  += w_fp * x_fp                 # tensor-core MMA

Key observations that will recur through 11_marlin_machete.md:

  1. Weight bytes/element = 0.5 B (INT4) vs 2 B (fp16). 4× less HBM traffic for weights.

  2. Activations still fetched at fp16. In W4A16 the activation cost dominates as batch grows — which is why activation quant becomes necessary at high batch.

  3. Dequant is register-local, essentially free if the kernel pipelines it well. That’s what Marlin nails.

  4. Zero-point handling is expensive if it’s asymmetric — you pay an extra sub. Marlin uses symmetric weights to skip this.

6. The three failure modes of naive affine quant

6a. One outlier destroys resolution

If a channel’s max is 100× the median, your s is set by the outlier, and 99% of your codepoints go unused. Consequence: naive per-channel INT4 on LLM activations sends PPL from 6 → 300+. All of 02_outlier_problem.md is about this.

6b. Rounding is not the only error

Naive round-to-nearest ignores the fact that not all weights matter equally. GPTQ (Hessian-weighted rounding), AWQ (channel-importance scaling), and NF4 (quantile-optimal codebook) all attack this from different angles.

6c. Zero-shift + numerical drift

For asymmetric quant, q - z on INT4 requires you to do the subtraction in an int type wide enough to hold negative values. Sloppy kernel writers use unsigned INT4 and get silent wraparound. Verify.

7. Formats you must be able to draw from memory

Before leaving this file, write these on paper without looking:

  1. FP16 (IEEE 754 binary16): 1 sign | 5 exp bias 15 | 10 mantissa — max ≈ 65504, has subnormals.

  2. BF16: 1 sign | 8 exp bias 127 | 7 mantissa — same range as fp32, less precision. Training darling.

  3. FP8-E4M3: 1 | 4 exp bias 7 | 3 mantissa — max ≈ 448 (with special-case saturating format), no inf in the OCP variant (only NaN). The inference default.

  4. FP8-E5M2: 1 | 5 exp bias 15 | 2 mantissa — max ≈ 57344. More range, less precision. Used for gradients / accumulators in FP8 training.

  5. FP4-E2M1: 1 | 2 exp bias 1 | 1 mantissa — 16 codepoints total. Values: {0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6}. This is what NVFP4 and MXFP4 both use for the data type; they differ in the scale type (FP8 vs E8M0).

  6. INT4 (signed): [-8, 7], 16 codepoints, equally spaced.

  7. NF4: 16 codepoints spaced at quantiles of N(0,1). Non-uniform. Symmetric around 0. Value set is a hardcoded lookup table — look it up in bitsandbytes source.

8. Practice exercises

Do these on paper before opening the next file:

  1. A weight w = 0.837 is quantized symmetrically with α = 1.0, INT4. Compute q. Then dequantize. What’s the rounding error?

  2. A per-channel scale for a Llama-3-8B down_proj weight ([4096, 14336]) at group=128 asymmetric INT4: how many bytes of scales+zeros? What’s the effective bits/weight?

  3. Convert x = 2.7 to FP8-E4M3. (Hint: exponent 1, biased exp = 8 = 1000, mantissa bits encode 1.35 — round to nearest representable.)

  4. NVFP4 stores group=16 with FP8-E4M3 scale + per-tensor FP32 scale. Draw the packed layout of one 16-element block. What’s the total bytes per block?

  5. Given a 4096×4096 weight quantized to symmetric per-channel INT4 with fp16 scales, how many bytes does the kernel fetch from HBM per output tile of 128 outputs (assuming activations are fp16 and 128 wide)?

Answer key at bottom of 14_projects.md.

Next: 02_outlier_problem.md — why none of this works out of the box on real LLM activations, and the entire lineage of fixes.