05 — AWQ: Activation-Aware Weight Scaling¶

Paper: AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — Lin, Tang, Tang, Yang, Xiao, Dang, Gan, Han (MIT HAN Lab). arxiv <phone_number_or_numberic_id_or_random_id_123> (Jun 2023, updated through 2024). Best-paper award at MLSys 2024.

Repos:

Reference: github.com/mit-han-lab/llm-awq (ships with the TinyChat runtime for edge deployment).
Community: github.com/casper-hansen/AutoAWQ — the operational default.
Production: also in github.com/vllm-project/llm-compressor.

Status in 2026: Alongside GPTQ as one of the two dominant W4A16 methods. Simpler math than GPTQ, no Hessian, and it tends to be more robust below 4-bit (AWQ 3-bit typically beats GPTQ 3-bit).

1. The single insight¶

Not all weights are equally important. About 1% of weights — the ones connected to the largest-magnitude activation channels — dominate model quality. Protect those channels; quantize the rest aggressively.

That’s the entire method. Everything else is executing on that idea.

The evidence¶

AWQ’s central experiment: pick a Llama-class model, quantize to INT4, but skip the 0.1% – 1% of weight channels whose input activation has the largest average magnitude (keep those in fp16). Result: PPL drops by 5–10× compared to naive INT4. Almost the entire quality loss from INT4 quant is concentrated in those few salient channels.

2. Why not just keep 1% in fp16?¶

Because a mixed-precision matmul (INT4 backbone + fp16 side-path for 1% of channels) is a nightmare for kernels. Every existing tensor-core GEMM assumes uniform precision. Mixed-precision splits the matmul, forces synchronization, breaks fusion. This is exactly the trap LLM.int8() fell into.

So AWQ asks: can we “protect” those channels while still keeping everything at INT4?

Yes — by scaling.

3. The scaling trick¶

Matrix multiplication has an invariance:

Y = X · W  =  (X · diag(s)⁻¹) · (diag(s) · W)   for any positive s

So you can rescale the input activations by dividing by s, and multiply the corresponding weights by s, and the output is mathematically identical. This is exactly the same invariance SmoothQuant uses (see 06_smoothquant.md).

AWQ says: choose s per input channel such that:

For the salient channels (large activations), s > 1, so the weights s · W become larger along that channel. Larger weight magnitudes mean the quantization noise is a smaller relative error.
For the non-salient channels, s ≈ 1.

And — crucially, unlike SmoothQuant — AWQ keeps activations at fp16 (it’s a W4A16 method). So there’s no downside from the X · diag(s)⁻¹ side.

Why does scaling reduce quant error?¶

For a symmetric per-channel scheme with scale α and INT4 codes:

Quant error on weight w is ≤ α/2.
Relative error is α/(2|w|).
Scaling w → s·w (with s > 1) doesn’t change α proportionally if you fix the per-group scale range — because with a fixed clip percentile, larger weights just fill more of the codebook.

More precisely (from the paper): scaling by s reduces expected error on that channel by ~s in the low-bit regime, and it works because the loss is dominated by the salient channels, which get the biggest s.

4. How to find `s` — the grid search¶

AWQ doesn’t derive s from a Hessian. It does something startlingly simple:

Compute per-channel activation magnitude |X_j| from a small calibration set (~128 samples, same shape as GPTQ).
Parameterize s_j = |X_j|^α · const_j, where α ∈ [0, 1] is a single scalar to search.
Grid search α over ~20 values from 0 to 1. For each, apply the scaling, quantize, measure MSE against fp16 output. Pick the best.

This takes minutes, not hours. There’s no Hessian, no Cholesky, no calibration-set dependence beyond the per-channel activation statistics.

Optional: per-channel clipping¶

After picking α, AWQ also grid-searches a per-channel clipping ratio (0.5 – 1.0 of max). Small further gain.

5. Bit-level output format¶

AWQ’s storage layout for a [out, in] weight in W4-group128 is very similar to GPTQ’s:

qweight: packed INT4, shape [in, out/8], in int32. (Note the transpose vs GPTQ: AWQ groups along output dim.)
qzeros: packed INT4 zeros, [in/g, out/8].
scales: fp16, [in/g, out].

Effective bits/weight: 4 + 16/g + 4/g = 4.156 bits at g=128.

The packing layout difference (input-major vs output-major) means AWQ and GPTQ checkpoints are not directly interchangeable, though both are readable by Marlin / Machete kernels via different launchers. llm-compressor unifies the on-disk format for vLLM consumption.

6. AWQ vs GPTQ — practical comparison¶

Axis	AWQ	GPTQ
Calibration cost	~5 min for 7B	~30–60 min for 7B (~4 GPU-hours for 175B)
Compute intensity	CPU-friendly bulk of work	GPU-required Cholesky
Robustness below 4-bit	Better (3-bit often usable)	Worse without extensive tuning
Robustness on unusual models	Better — the scaling doesn’t need Hessian PSD assumptions	Occasional numerical instability
Best PPL at W4-group128 (Llama-3-8B)	~6.24–6.30	~6.20–6.30
Best PPL at W3-group128	~7.0–7.4	~7.5–8.5
Ecosystem coverage	Excellent — HF + vLLM + TinyChat + edge	Excellent — HF + vLLM + more variants
Multi-modal support	Native in `llm-awq` (VLM-friendly)	Retrofit-y

Practical guidance: for W4-group128 on a well-studied Llama/Qwen/Mistral, they’re a coin flip — use whichever your production stack (vLLM, TensorRT-LLM, etc.) has the fastest kernel for. In practice this is currently Marlin, which supports both, and Machete on Hopper, which supports both.

For 3-bit or unusual models (Chinese-language, tool-heavy, small): try AWQ first.

For easy reproducibility and academic pedigree: GPTQ has more downstream methods citing it.

7. AWQ + the outlier story¶

Re-read 02_outlier_problem.md. AWQ sits in the “protect the outlier channels” family, but with a twist: it doesn’t isolate them in fp16 (like LLM.int8()) or migrate them (like SmoothQuant). Instead it amplifies them within the quantization codebook by scaling the salient channels up, so their INT4 codes have more distinguishing precision.

Elegant. Cheap. Kernel-friendly. That’s why it won MLSys best paper.

8. AWQ’s descendants¶

A few methods explicitly build on AWQ’s scaling idea:

QServe (QoQ) — arxiv <phone_number_or_numberic_id_or_random_id_124>, mit-han-lab/omniserve. AWQ-style scaling extended to W4A8KV4 with a companion kernel that solves the CUDA-core dequant bottleneck. If you care about batch-32+ serving with 4-bit weights, this is the current frontier.
OmniQuant — arxiv 2308.13137. Makes the AWQ scaling factor learnable (gradient-descent on s values), plus similar learnable transformations on activations. Better numbers at 3-bit, more compute.
AWQ + Marlin. Not a new method, but the operational thing: AWQ-quantized weights, Marlin kernel. See 11_marlin_machete.md.

9. How to run AWQ in 2026¶

Option A: `AutoAWQ`¶

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

quant_config = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMM",   # "GEMM" for Marlin-compatible; "GEMV" for TinyChat edge
}

model = AutoAWQForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
tok   = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
model.quantize(tok, quant_config=quant_config)
model.save_quantized("llama3-8b-awq-w4")
tok.save_pretrained("llama3-8b-awq-w4")

Option B: `llm-compressor`¶

from llmcompressor.modifiers.awq import AWQModifier

recipe = AWQModifier(
    scheme="W4A16",
    bits=4,
    group_size=128,
    targets="Linear",
    ignore=["lm_head"],
)
# oneshot(...) as in GPTQ example in 04_gptq.md

Both produce vLLM-ready checkpoints. llm-compressor is preferred if you’re going to production; AutoAWQ is a bit more flexible for experiments.

10. Anti-patterns to avoid¶

Calibrating AWQ on eval set. Same warning as GPTQ. Don’t.
Using AWQ on a base model when you deploy the instruct variant. The activation statistics differ enough that PPL degrades. Calibrate on the exact checkpoint you deploy.
Ignoring lm_head. By default AWQ (and GPTQ) skip lm_head — it should stay fp16. If you quantize it, quality dies. This is not obvious; check your config.
Using the GEMV variant on server. GEMV is for TinyChat (batch 1, edge). For server, GEMM. This is the version config field.

11. study one-liner¶

AWQ observes that ~1% of weight channels — those connected to the largest-magnitude activation channels — dominate LLM quality. Instead of keeping those in fp16 like LLM.int8() (which breaks kernel fusion), AWQ exploits the matmul scaling invariance to scale those salient channels’ weights up by a per-channel factor derived from activation magnitude. Larger weight magnitudes take a smaller relative quantization error, so those channels are effectively protected while everything stays uniformly INT4. Search over one scalar α on 128 calibration samples; done in five minutes.

Next: 06_smoothquant.md — the same scaling invariance, but used to enable W8A8 instead of W4A16.