Phase 5 — Quantization & Model Compression

Goal: Become the person who knows which 4 bits to keep.

Duration: Months 8–10 in the 13-month arc (overlaps end of Phase 4 engine work, sets up Phase 6 distributed). Prereqs: Phase 2 GPU arch (you understand HBM bandwidth ↔ tensor-core FLOPs), Phase 3 kernel work (you can read a Triton kernel), Phase 4 engine internals (you know why decode is memory-bound).

Why this phase matters to you specifically

You are an applied-ML engineer at Zoho with on-prem customer exposure. Every on-prem deployment is a quantization problem in disguise:

  • The customer has 2×L40S / 4×L4 / 1×A100 — not 8×H100. Model choice is bounded by VRAM.

  • Data can’t leave the building → no API fallback. Quality regression from quantization is your production incident.

  • Batch profiles are lumpy: agentic loops fire bursts (compute-bound prefill on shared prompts) then long idle (memory-bound decode at batch 1).

  • Latency SLOs are user-facing (chat CRM assistant, ticket summarization).

The engineer who can walk in and say “on 2×L40S, Qwen2.5-32B in W4A16-Marlin with fp8 KV gets you 45 tok/s at batch 1 and PPL degradation of 0.08 vs bf16, whereas naive AWQ-INT4 on generic kernels gets you 12 tok/s” — that engineer is who Zoho pays enterprise consulting fees to. That is the target profile of this phase.

The philosophy: “Which 4 bits to keep”

Everything in modern LLM quantization reduces to a single question:

Given N bits, where do you spend them?

The naive answer is “uniformly, on every weight.” The correct answer is “non-uniformly, biased toward the ~0.1–1% of weights and activations that matter, and away from everything else.”

That single reframing generates the entire modern method zoo:

Method

“Where to spend bits” answer

GPTQ

Spend precision on weights that lie on high-curvature directions of the Hessian

AWQ

Protect the ~1% weight channels connected to salient activation channels

SmoothQuant

Migrate activation outliers into weights so both quantize cleanly at 8-bit

QuaRot/SpinQuant/DuQuant

Rotate the space so no channel is an outlier — spread evenly

KIVI

Key cache has channel-outliers → quant per-channel; value cache doesn’t → per-token

GGUF k-quants

Super-block + block scales + importance matrix (imatrix) prioritize “important” weights

NF4 (QLoRA)

Bit codes designed for zero-mean-unit-variance normal weight distributions

BitNet

Push the question to training: teach the model to live at 1.58 bits

The through-line: outliers eat your bits. Every advance since 2022 is a smarter way to manage the ~1% of values that would otherwise dominate your quantization error budget.

The one asymmetry rule you must internalize

Memorize this. It is the study signal in this field:

W4A16 wins at low batch (memory-bound decode). W8A8 / FP8 wins at high batch (compute-bound prefill).

Why: decode at batch 1 streams every weight through HBM to produce a handful of tokens — bandwidth is the wall, so shrink the weights (4-bit) and pay the dequant cost in registers (~free). Prefill at long prompts, or decode at batch 32+, saturates tensor cores — bandwidth stops being the wall, so use native low-precision compute (int8/fp8 tensor cores, which are 2×/4× the fp16 throughput on Hopper/Blackwell).

Full worked roofline math in 03_asymmetry_rule.md. This asymmetry falls straight out of the roofline you already learned in Phase 2. If it doesn’t feel obvious yet, go back and re-derive it — every subsequent decision in this phase depends on it.

The reading order (do it in this sequence)

  1. 01_quant_theory.md — Affine quant, scale/zero-point, symmetric vs asymmetric, granularity ladder. The vocabulary you’ll speak all phase.

  2. 02_outlier_problem.md — LLM.int8(), massive activations. All later methods make sense only after you feel this pain.

  3. 03_asymmetry_rule.md — The W4A16 vs W8A8 roofline decision, with worked math.

  4. 04_gptq.md — OBS → OBQ → GPTQ. Understand the Hessian intuition; recent proof (arxiv 2507.18553) shows GPTQ ≡ Babai’s nearest-plane algorithm.

  5. 05_awq.md — Activation-aware channel scaling; the “protect 1%” idea.

  6. 06_smoothquant.md — Migrating difficulty to weights for W8A8; still the production W8A8 baseline.

  7. 07_fp8_family.md — E4M3/E5M2, per-tensor / per-channel / per-block scaling, DeepSeek-V3 as the FP8 training existence proof.

  8. 08_fp4_mxfp4_nvfp4.md — Blackwell-era. OCP MX spec, NVFP4 group-of-16, MXFP4 group-of-32-with-E8M0. The 2025 story.

  9. 09_kv_cache_quant.md — Where KIVI and KVQuant win, and why fp8 KV is the free-win default.

  10. 10_gguf_k_quants.md — llama.cpp bit layouts, super-block structure, i-quants, imatrix.

  11. 11_marlin_machete.md — The kernel that makes W4A16 usable. Ampere vs Hopper.

  12. 12_evaluation.md — Perplexity, lm-eval, KL-div, long-context, “no eval = vandalism.”

  13. 13_bake_off_project.md — The portfolio project. 8B model × 4 formats × 2 batch sizes × 4 metrics.

  14. 14_projects.md — All other Phase 5 hands-on projects.

Rough calendar (Months 8–10, ~40 hours/month)

Week

Reading + writing

Doing

1

01, 02, 03 (theory + outliers + rule)

Implement affine quant in numpy; visualize activation histograms of a real Llama layer; draw the outlier

2

04, 05 (GPTQ, AWQ)

Quantize a 1B model with AutoGPTQ and llm-compressor; verify PPL sanity

3

06, 07 (SmoothQuant, FP8)

Run FP8 in vLLM (--quantization fp8); compare vs bf16 on your rig

4

08 (FP4 landscape)

Read MR-GPTQ paper; read Blackwell whitepaper; if no B200 access, at least read the CUDA code

5

09 (KV cache)

Enable fp8 KV in vLLM; measure long-context PPL delta

6

10 (GGUF)

Write a Python parser for GGUF headers; dump a Q4_K_M block to hex; do llama.cpp bake-off

7

11 (kernels)

Read Marlin repo top-to-bottom; write a naive Triton W4A16 GEMM

8

12 (eval)

Wire up lm-eval-harness + KL harness; contaminate-free WikiText-2 PPL script

9–12

13, 14 (project)

The bake-off project. Ship it. Post it.

Exit criteria

You have finished Phase 5 when all four of these are true:

  1. Prescription: Given a deployment scenario — hardware, batch profile, latency SLO, quality tolerance — you can prescribe a quantization scheme (weight format + activation format + KV format + which kernel) and justify it with roofline math + outlier reasoning in ≤10 minutes on a whiteboard.

  2. Numerics: You can explain, at bit-level, the layout of Q4_K_M, NF4, MXFP4, NVFP4, and FP8-E4M3. You can also convert a real number to each and back.

  3. Public artifact: The bake-off write-up (13_bake_off_project.md) is published. r/LocalLLaMA post included. Quality tables and speed tables and KL-divergence and long-context spot check.

  4. Kernel reality check: Your naive Triton W4A16 kernel exists in a repo. You have measured its gap vs Marlin/Machete and written a paragraph explaining every reason for the gap (occupancy, dequant scheduling, tensor-core layout, SMEM bank conflicts).

If any of those four is missing — you haven’t finished the phase. This is the vandalism guardrail.

What’s not in this phase (deliberately)

  • QAT (Quantization-Aware Training): Awareness only. It requires a training pipeline, which is Phase 6. Read the BitNet papers (2411.04965, 2504.18415) and the QAT scaling law paper (2505.14302) — but don’t try to train QAT models yet.

  • Distillation & pruning: Awareness in 14_projects.md. Deep coverage would take another phase and isn’t the r/LocalLLaMA-facing skill you’re building.

  • Hardware-specific compilation (TensorRT-LLM PTQ toolkit): Touched in Phase 7 (production). Here we stay in the PyTorch/vLLM/llama.cpp world where the community actually lives.

Canonical repos you will interact with

Verify all before use — the field ships weekly.

Repo

Purpose

Status (mid-2026)

github.com/IST-DASLab/gptq

Original GPTQ

Reference impl; superseded operationally by AutoGPTQ / llm-compressor

github.com/AutoGPTQ/AutoGPTQ

Community GPTQ toolkit

Widely used; slow-moving

github.com/casper-hansen/AutoAWQ

Community AWQ toolkit

Actively maintained

github.com/mit-han-lab/llm-awq

Reference AWQ + TinyChat

Research repo; use AutoAWQ for prod

github.com/mit-han-lab/smoothquant

Reference SmoothQuant

Frozen; concept lives in llm-compressor

github.com/mit-han-lab/omniserve

QServe W4A8KV4

Newer; watch it

github.com/vllm-project/llm-compressor

Neural Magic’s unified toolkit — GPTQ/AWQ/SmoothQuant/FP8/NVFP4/MXFP4/sparsity → vLLM-ready

The tool to know in 2026.

github.com/IST-DASLab/marlin

W4A16 kernel (Ampere)

Merged into vLLM

github.com/spcl/QuaRot

Rotation-based W4A4

Research; concept absorbed into llm-compressor

github.com/facebookresearch/SpinQuant

Learned rotations

Meta, Llama-oriented

github.com/ggerganov/llama.cpp

GGUF + k-quants + i-quants

The most-used LLM runtime on the planet

github.com/EleutherAI/lm-evaluation-harness

Task evals

Standard; use v0.4+ API

The 6-line mental cheatsheet

Print this. Stick it above your monitor.

  1. Decode = memory-bound → weight-only 4-bit (W4A16).

  2. Prefill / big batch = compute-bound → activation quant (W8A8, FP8, eventually FP4).

  3. Outliers eat your bits — every method is an outlier-management strategy.

  4. KV cache to fp8 is nearly free; int4 KV needs KIVI-style per-channel-key / per-token-value.

  5. Marlin/Machete = the reason W4A16 is fast at batch >1; naive kernels fall off a cliff.

  6. No eval = vandalism. Every quant claim ships with PPL + task evals + KL + long-context.

Now go to 01_quant_theory.md.