04 — Quantization Papers¶

Phase alignment: Months 11–14 (Phase 5, overlapping Phase 4 late stage).

This set is the shortest and the most bit-level. Every paper answers one question: which bits do we keep? The answer changes depending on whether you’re memory-bound (decode) or compute-bound (prefill, high-batch), and which outlier problem you’re fighting.

1. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale¶

arxiv:<phone_number_or_numberic_id_or_random_id_39> · Dettmers et al., NeurIPS ‘22 · [MEDIUM] · Prereqs: transformer basics, floating-point representation

Key trick: Split each matmul into two parts — the ~0.1% of columns with outlier activations get computed in fp16, everything else in int8 — recovering fp16 quality at 2× memory savings.

What to extract:

The outlier discovery: at scale (>6.7B), a small set of feature dimensions develop activation magnitudes 100–1000× the rest. Quantize them naively and quality collapses.
Mixed-precision decomposition: outlier columns go through fp16 GEMM, non-outliers through int8, results summed. Overhead ~5%.
This paper is the origin of “activation outliers are the problem.” Every subsequent method (SmoothQuant, AWQ, FP8 per-block) is an outlier-management strategy. Do not read them until you feel this.
The bitsandbytes library is Dettmers’s implementation. It’s slow — the ideas matter, the code is a starting point.

Read alongside: “Massive Activations in Large Language Models” (arxiv:<phone_number_or_numberic_id_or_random_id_40>) — later work characterizing where outliers appear (specific tokens, specific channels, deep layers). Two-hour read, changes how you look at quantization.

2. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers¶

arxiv:<phone_number_or_numberic_id_or_random_id_41> · Frantar, Ashkboos, Hoefler, Alistarh · ICLR ‘23 · [HARD] · Prereqs: #1, basic optimization theory

Key trick: Quantize weights column by column, using the inverse Hessian of the reconstruction loss on calibration data to update remaining unquantized columns to compensate for the rounding error — 4-bit weights with fp16-level task quality.

What to extract:

The Optimal Brain lineage: OBS (1993) → OBD → OBQ (Frantar 2022) → GPTQ. Each simplifies the Hessian assumptions to scale up.
Why calibration data matters: the Hessian is computed on a small calibration set (~128 sequences). The choice of calibration data affects quality (mildly for in-domain, more for out-of-domain).
Group-size 128 is the standard: one scale/zero per group of 128 weights in a row.
Weight-only, activations stay fp16. This is a W4A16 method — wins at low batch (memory-bound), doesn’t accelerate prefill.
Read AutoGPTQ code: https://github.com/AutoGPTQ/AutoGPTQ or the newer llm-compressor (https://github.com/vllm-project/llm-compressor).

3. AWQ: Activation-aware Weight Quantization¶

arxiv:<phone_number_or_numberic_id_or_random_id_42> · Lin et al., MLSys ‘24 · [MEDIUM] · Prereqs: #1, #2

Key trick: ~1% of weight channels are “salient” (they correspond to large-activation input channels). Protect them by scaling activations down and weights up by a per-channel factor s (mathematically neutral), so post-quant error concentrates on non-salient weights.

What to extract:

The scale-migration trick: y = W · x = (W · diag(s)) · (diag(1/s) · x). The math is invariant; the quantization error is not.
No backprop, no gradients, just per-channel scale search on calibration data (fast: minutes for 7B).
Search over a 1D grid per layer for optimal s. Simple and stable.
AWQ vs GPTQ empirical: AWQ usually slightly better quality at 4-bit, faster to compute, more robust to calibration data mismatch. GPTQ has more mature kernel support in some engines.
Marlin kernel (below) makes AWQ+GPTQ fast at batch >1.

4. SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs¶

arxiv:<phone_number_or_numberic_id_or_random_id_43> · Xiao et al., ICML ‘23 · [MEDIUM] · Prereqs: #1, #3

Key trick: Same scale-migration idea as AWQ but for W8A8 — push the difficulty of quantizing outlier activations onto the weights (which are static and quantization-friendly), so activations become quantizable at int8.

What to extract:

Why W8A8 matters that W4A16 doesn’t: activation quant enables int8 tensor-core paths for GEMM — wins at high batch and prefill, the compute-bound regimes.
Migration strength α (default 0.5): interpolates between all-weight (α=1) and all-activation (α=0) difficulty. Tune per model.
Combine with GPTQ for W4A8 (weights 4-bit, activations 8-bit).
The strategic framing: W4A16 (GPTQ/AWQ) for local/low-batch decode; W8A8 (SmoothQuant) or FP8 for high-throughput serving. You’ll invoke this framing in studies.

5. QLoRA: Efficient Finetuning of Quantized LLMs¶

arxiv:<phone_number_or_numberic_id_or_random_id_44> · Dettmers et al., NeurIPS ‘23 · [EASY] · Prereqs: LoRA, #1

Key trick: Freeze the base model in NF4 (4-bit NormalFloat, information-theoretically optimal for normal-distributed weights), attach LoRA adapters in bf16, backprop only through the adapters — fine-tune 65B on a single 48GB GPU.

What to extract:

NF4 quantization: 16 non-uniform levels chosen to maximize information under the assumption weights are ~N(0,1) after normalization. Contrast with uniform int4.
Double quantization: even the quantization constants get quantized. Saves ~0.4 bits/param.
Paged optimizers: NVIDIA unified memory to swap Adam states to CPU. Not a quantization idea, a memory-management one — read it.
QLoRA is your bridge to Phase 6 fine-tuning. NF4 is not a serving format (dequantize-on-the-fly is slow); it’s a training-time memory hack.

6. FP8 Formats for Deep Learning (Micikevicius et al.)¶

arxiv:<phone_number_or_numberic_id_or_random_id_45> · NVIDIA/Arm/Intel joint spec, 2022 · [EASY] · Prereqs: floating-point representation

Key trick: Standardize two fp8 formats — E4M3 (4 exp, 3 mantissa, range ±448, high precision) and E5M2 (5 exp, 2 mantissa, range ±57344, high dynamic range) — covering the twin needs of forward activations/weights (E4M3) and gradients (E5M2).

What to extract:

Draw both bit layouts. Compare to bf16 (1/8/7) and fp16 (1/5/10).
Per-tensor scaling (Hopper baseline): one scale per tensor, applied before/after tensor-core MMA. Requires calibration or dynamic tracking.
Per-block scaling (Blackwell/MXFP): scale per small block (e.g., 32 elements). Handles outliers better; adopted by DeepSeek-V3 training.
Hopper Transformer Engine’s automatic-scaling for FP8 training is worth understanding at API level.
FP8 tensor cores are ~2× fp16 TFLOPs on H100 (1979 vs 989 bf16). This is the compute reason to move to fp8.

7. OCP Microscaling Formats (MX) Specification¶

Spec, not paper. OCP, 2023 · https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf · [MEDIUM] · Prereqs: #6

Key trick: Standardize small-block scaled formats — MXFP8/6/4 and MXINT8 — where each block of 32 elements has a shared E8M0 (8-bit exponent) scale factor. Enables 4-bit floats with per-block outlier handling in hardware.

What to extract:

MXFP4 = E2M1 (2 exp, 1 mantissa) elements with E8M0 shared exponent per block of 32. 4.25 bits/element effective.
NVFP4 = NVIDIA’s variant using E4M3 scales instead of E8M0. Better precision, marginally more overhead.
Blackwell tensor cores natively support MX/NVFP4 GEMM. 5x fp8 peak (per NVIDIA marketing).
Read the spec once; it’s ~30 pages and clarifies which format winds up where.

8. KIVI: Plug-and-Play 2-bit KV Cache Quantization¶

arxiv:<phone_number_or_numberic_id_or_random_id_46> · Liu et al., ICML ‘24 · [MEDIUM] · Prereqs: #1

Key trick: K and V have different outlier structure — K has channel-wise outliers, V has token-wise outliers — so quantize K per-channel and V per-token to 2-bit with minimal quality loss, shrinking KV cache 8× vs fp16.

What to extract:

The asymmetry: K is compared against Q via dot products, so per-channel scaling aligns with the reduction axis. V is a weighted sum with softmax weights, so per-token quantization is natural.
Sliding-window: keep the most recent ~128 tokens in fp16 (needed for quality on recent tokens), quantize older to 2-bit.
Why KV quant matters more than weight quant for long context: at 128k context, KV dwarfs weights for a 70B model (do the math: ~40GB KV in fp16 vs 140GB weights).
FP8 KV is a cheaper, safer version; INT4 KV is where research is still shaking out; INT2 KV (KIVI) is the frontier.

9. Marlin: Fast 4-bit Weight Matrix Multiplication for Ampere GPUs¶

No paper (repo + writeup). https://github.com/IST-DASLab/marlin · [HARD] · Prereqs: CUDA fluency, papers #2, #3

Key trick: Weights stay 4-bit in HBM (bandwidth win), get dequantized in registers/shared memory right before the fp16 tensor-core MMA, fused in one Ampere-optimized kernel that finally hits GPU peak for W4A16 at batch >1.

What to extract:

Why AWQ/GPTQ were slow before Marlin: the naive path (dequantize → cuBLAS GEMM) has to write fp16 weights to HBM, costing more than the bandwidth savings. Marlin does it all in one kernel.
Async pipelining, shared-memory double buffering, warp-specialized fetch/compute, StreamK-like load balancing.
The repo’s README is a masterclass in Ampere kernel optimization. Read it word by word.
Batch-size scaling: Marlin is ~cuBLAS-fp16 at batch 1 (memory-bound), and stays close as batch grows (compute-bound with dequant overhead hidden).

Successor: Machete (https://github.com/vllm-project/vllm/tree/main/csrc/quantization/machete) — Hopper version using wgmma. Study after Marlin.

10. GGUF k-quants and i-quants (llama.cpp)¶

No paper. Docs: https://github.com/ggml-org/llama.cpp/wiki/Tensor-Encoding-Schemes · [MEDIUM] · Prereqs: quantization basics

Key trick: Block-wise mixed-precision quantization designed for CPU and consumer GPU inference: super-blocks of 256 weights with a 6-bit master scale, sub-blocks of 16–32 with individual 4/5/6-bit scales, and importance-weighted variants (imatrix) that use activation statistics.

What to extract:

Q4_K_M = the local-LLM default: 4-bit weights, super-block structure, ~4.5 bits/param effective, ~fp16 quality within 1%.
IQ4_XS / IQ3_M / IQ2_XXS = importance-weighted variants; use a calibration matrix (imatrix) to protect salient weights. 3-bit becomes usable.
The GGUF file format spec (https://github.com/ggml-org/ggml/blob/master/docs/gguf.md) — read it and write a Python parser for the header. Under 100 LOC.
Quantize a model yourself: llama.cpp/convert-hf-to-gguf.py then llama-quantize. Bench with llama-perplexity.

The whiteboard test¶

At the end of Phase 5 you should be able to:

Pick a quantization scheme given a deployment scenario. Example: “70B model, 4×A100 40GB, batch 32, chat SLA 100ms ITL.” Answer with format + reasoning (W8A8 fp8 for compute, GQA already saves KV, fp8 KV as second layer).
Explain why W4A16 wins at low batch and W8A8 wins at high batch in terms of roofline.
Derive how much KV memory a 70B model uses at 128k context in fp16, fp8, and int4.
Compare GPTQ vs AWQ vs SmoothQuant in one sentence each.
Draw the bit layout of E4M3 and Q4_K_M super-block from memory.

The project that closes the section (also in Phase 5 exit criterion)¶

One 7–8B model → produce GPTQ-INT4, AWQ-INT4, FP8 (llm-compressor), GGUF Q4_K_M variants → benchmark all on:

Speed batch 1 and batch 32 (vllm bench serve for GPU, llama-bench for GGUF)
Quality perplexity on WikiText-2 + 3 tasks from lm-evaluation-harness + KL divergence vs fp16 on a 1000-prompt sample

Publish the table as a blog post on your GitHub Pages / Substack. Cross-post to r/LocalLLaMA. This is the paper’s test.