04 — GPTQ: Hessian-Weighted Rounding, the OBS Lineage, and Why It Still Ships¶

Paper: GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Frantar, Ashkboos, Hoefler, Alistarh (IST-DASLab). arxiv <phone_number_or_numberic_id_or_random_id_119> (Oct 2022).

Repo: github.com/IST-DASLab/gptq (reference). Prod: github.com/AutoGPTQ/AutoGPTQ and github.com/vllm-project/llm-compressor.

Status in 2026: Still one of the two dominant W4A16 methods (with AWQ). Both are usually run through the same llm-compressor pipeline and paired with the Marlin kernel. Recently (arxiv <phone_number_or_numberic_id_or_random_id_120>, 2025) proven to be exactly equivalent to Babai’s nearest-plane CVP algorithm — which improves it further.

1. The problem GPTQ solves¶

Given a weight matrix W and a small calibration set of activations X (say, 128 samples of 2048 tokens from WikiText or C4), find quantized weights Ŵ that minimize:

||WX - ŴX||²_F

i.e., the layer output error, not just ||W - Ŵ||. This is the crucial reframing — nothing about naive round-to-nearest cares about how the activations interact with the weights.

2. The lineage: OBS → OBD → OBQ → GPTQ¶

Optimal Brain Damage (OBD) — LeCun, Denker, Solla (1990)¶

A pruning method. For a trained network at a loss minimum, use second-order Taylor to predict the loss increase from setting weight w_i = 0:

δL_i ≈ (1/2) · w_i² · H_ii

Delete the weights with smallest predicted increase. Innovation: the Hessian tells you which weights are cheap to lose.

Optimal Brain Surgeon (OBS) — Hassibi, Stork (1993)¶

OBD assumes H is diagonal. OBS uses the full Hessian: when you set w_i = 0, you should also update the remaining weights to compensate. The optimal update is:

Δw = -(w_i / [H⁻¹]_ii) · H⁻¹·e_i

and the resulting loss increase is w_i² / (2 · [H⁻¹]_ii). Innovation: you don’t just delete — you compensate by shifting other weights.

OBQ (Optimal Brain Quantization) — Frantar & Alistarh (2022)¶

Same idea, quantization instead of pruning. For each weight, compute the quantization error δ = w_i - quant(w_i), then apply the OBS update to the unquantized weights to compensate.

Problem: H⁻¹ is expensive. Even with the LLM trick (H = 2·X·Xᵀ for layer input X), OBQ’s per-weight update rule is O(N²) per weight and O(N⁴) total. Fine for 100M-param networks. Impossible for GPT-3-175B.

GPTQ — the 2022 breakthrough¶

Three modifications, all pragmatic:

Process weights in a fixed order (columns of W, back to front) rather than greedily picking the “cheapest” one. Sub-optimal on paper, ~free on GPU (removes an argmin).
Lazy batching of Hessian updates: apply many updates in a batch, use blocked Cholesky, get everything into cuBLAS.
Cholesky reformulation: instead of maintaining H⁻¹ and updating it (unstable), use one Cholesky decomposition of H⁻¹ upfront and read the rows off in order.

Result: quantize an OPT-175B in ~4 GPU-hours. Layer-output MSE nearly as low as full OBQ. Almost no PPL degradation at 4-bit (Llama-3-8B: fp16 PPL 5.6 → GPTQ-4bit 5.7–5.8 on WikiText-2 with group-128).

3. The intuition in one paragraph¶

A weight sits at value w. Naive round-to-nearest sends it to q ≈ round(w/s), incurring error δ = w - s·q. GPTQ observes: that error can be partly absorbed by nudging the not-yet-quantized weights. The nudge is dictated by the (inverse) Hessian, which encodes the calibration-activation-derived importance and covariance of each weight direction. In practice this means: process weights left-to-right; each time you round a weight, subtract a Hessian-weighted correction from the remaining columns. By the time you get to the last column, all the accumulated error has been squeezed into places the Hessian says it will do least damage.

4. The updated 2025 view: GPTQ = Babai’s nearest-plane algorithm¶

Paper: GPTQ Reconsidered — arxiv <phone_number_or_numberic_id_or_random_id_121> (2025). Repo: github.com/IST-DASLab/GPTQ-Babai.

Main result: when GPTQ processes columns in reverse order (back-to-front, as in the reference impl), it is literally equivalent to Babai’s nearest-plane algorithm for the Closest Vector Problem (CVP) on the lattice defined by the calibration Hessian.

Why this matters:

Error bound: Babai’s algorithm has a provable approximation ratio. Now GPTQ inherits it — previously the error was empirical.
Fixes clip failure: the original GPTQ often disabled clipping to avoid a numerical instability. The Babai view shows that instability is an accidental interaction between the Cholesky and the clip; the reconsidered version handles clipping cleanly and gets consistently better numbers.
Weight ordering matters: the paper shows that pre-permuting columns by activation-norm importance yields a further ~0.1–0.3 PPL gain. This is what act_order=True in AutoGPTQ has always been — now theoretically justified.

Bottom line for practice: use act_order=True and use the llm-compressor GPTQ path. It quietly incorporates these fixes.

5. Calibration set: how much does it matter?¶

Empirical answer, memorize it:

Size: 128 samples × 2048 tokens = ~262K tokens is the sweet spot. More helps marginally; less hurts.
Source: Match the target domain when possible. For a code model, calibrate on code (StarCoder set). For a chat model, calibrate on OpenOrca or ShareGPT-style data. Using WikiText for a code model gives you visibly worse code PPL.
Sequence length: 2048 is enough. Longer doesn’t help GPTQ specifically (it helps AWQ marginally).
Sensitivity: for W4-group128, the calibration-set choice moves PPL by ~0.05–0.15. Non-trivial, not catastrophic. For W3 and W2 it becomes huge (~0.5–2 PPL). Below 4 bits, calibration is critical.

Anti-pattern: calibrating on the eval set. Yes, people do this. It looks great on the leaderboard. It’s contamination. Don’t.

6. Where GPTQ falls down¶

Activation quant — GPTQ doesn’t do it. It’s weight-only. For W8A8/W4A8, you need SmoothQuant or QServe (which sits on top of GPTQ).
Very-narrow-width failure: at 3-bit and below, GPTQ needs act_order, careful damp_percent, and a lot of hand-tuning. AWQ tends to be more robust below 4-bit.
Calibration mismatch: if you calibrate on English Wikipedia and deploy on Bengali, you can see 1–2 PPL delta beyond just the language shift. GGUF’s imatrix flow has the same issue.
No handling of massive activations: GPTQ minimizes layer-output MSE assuming your calibration activations are representative. If a rare token (BOS, delimiter) produces a massive activation, GPTQ’s Hessian doesn’t see it and you can get quality regressions in edge cases.

7. How to run GPTQ in 2026¶

Don’t clone the original repo. Use one of:

Option A (recommended): `llm-compressor`¶

from llmcompressor.transformers import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier

recipe = GPTQModifier(
    targets="Linear",
    scheme="W4A16",             # or "W8A16", etc.
    ignore=["lm_head"],
    dampening_frac=0.01,
    actorder="weight",           # the 2025 fix, uses Babai-style ordering
)

oneshot(
    model="meta-llama/Llama-3.1-8B-Instruct",
    dataset="open_platypus",     # or your own
    recipe=recipe,
    output_dir="llama3-8b-w4a16-gptq",
    max_seq_length=2048,
    num_calibration_samples=512,
)

Produces a vLLM-ready checkpoint. Ships with Marlin kernel auto-selection. This is the production path.

Option B: `AutoGPTQ`¶

More knobs, older API, still widely used in the HF ecosystem. Good for reproducibility of pre-2025 papers.

8. What GPTQ output looks like at bit level¶

A GPTQ W4-group128 checkpoint for a linear layer of shape [out=4096, in=4096] stores, per shard/tile that vLLM/Marlin can read:

qweight: packed INT4 in an int32 tensor of shape [in/8, out]. 8 INT4 values per int32.
qzeros: packed INT4 zero-points, one per group, of shape [in/g, out/8].
scales: fp16 scales, shape [in/g, out]. (g=128.)
g_idx: an int32 vector of length in, mapping each input dim to its group (for act_order=True, this is a permutation).

Total bytes ≈ out·in/2 (weights) + out·in/g · 2 (scales) + out·in/(g·2) (zeros) + in·4 (g_idx). For 4096×4096, that’s ~8.4 MB — 4.10 bits/weight effective.

The kernel side (Marlin) fetches all of this in a specific tiled order and dequantizes just before the tensor-core MMA. Covered in 11_marlin_machete.md.

9. Quality expectations (Llama-3-8B, WikiText-2 PPL, real numbers)¶

Format	PPL	Notes
fp16 baseline	6.14
GPTQ W8A16 group=-1 (per-channel)	6.14–6.15	Free win
GPTQ W4A16 group=128	6.24–6.35	Ships everywhere
GPTQ W4A16 group=128 + act_order	6.20–6.30	Slightly better
GPTQ W3A16 group=128	6.9–7.5	Painful without careful tuning
GPTQ W2A16 group=64	15–30+	Do not use — use AQLM/QuIP# instead below 3-bit

For 70B models the pattern holds but with tighter margins (fp16 → W4 delta ~0.05 PPL, ~1% relative).

10. study one-liner¶

GPTQ is Hessian-weighted rounding: instead of round-to-nearest, it processes weights column by column and after quantizing each weight, applies a compensating update to the remaining unquantized weights based on the inverse Hessian of the calibration activations. That way, when you’re forced to round a weight up, you can nudge the others down to preserve the layer’s output. Recent theory (arxiv <phone_number_or_numberic_id_or_random_id_122>) proves the algorithm is Babai’s nearest-plane on the lattice defined by that Hessian, which gives it a provable error bound and clean handling of clipping.

Next: 05_awq.md — the other dominant W4A16 method. Simpler, faster to compute, no Hessian, protects a different set of weights.