06 — SmoothQuant: Migrating Difficulty from Activations to Weights

Paper: SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models — Xiao, Lin, Seznec, Wu, Demouth, Han (MIT HAN Lab, NVIDIA). arxiv <phone_number_or_numberic_id_or_random_id_113> (Nov 2022).

Repo: github.com/mit-han-lab/smoothquant (reference, frozen). Production: github.com/vllm-project/llm-compressor and github.com/NVIDIA/TensorRT-LLM calibration paths.

Status in 2026: Still the baseline W8A8 method. Superseded for W4A4 by rotation methods (QuaRot etc.), but for W8A8 it is the workhorse and the concept lives on inside QServe (W4A8KV4).

1. The problem

Recall from 02_outlier_problem.md: LLM activations have channel outliers 10–100× the median. A per-tensor INT8 quant of activations sets the scale from those outliers, destroys resolution for the bulk, and PPL explodes.

But — weights of the same LLMs are well-behaved. Roughly normal, low kurtosis, no dominant channels. Per-channel INT8 quantization of weights is essentially lossless.

SmoothQuant asks: can we redistribute the difficulty?

2. The same invariance, used the other direction

Recall the AWQ trick (05_awq.md):

Y = X · W  =  (X · diag(s)⁻¹) · (diag(s) · W)

AWQ used this to make certain weights larger (better relative INT4 error). SmoothQuant uses it in the opposite direction:

  • Divide the activations X by s (per input channel) → smooths the activation outliers.

  • Multiply the weights W by s (per input row) → makes weights harder, but only marginally (weights start easy).

Choose s such that the combined quant error (activation error + weight error) is minimized. This is a W8A8 method — both weights and activations end up in INT8.

3. How to choose s

The paper parameterizes:

s_j = max(|X_j|)^α / max(|W_j|)^(1-α)

where j indexes input channels, and α ∈ [0, 1] is a scalar. Intuition:

  • α = 0: s_j = 1 / max(|W_j|) — all difficulty in activations (no migration).

  • α = 1: s_j = max(|X_j|) — fully flatten activations (weights take all the pain).

  • α = 0.5: split the load.

In the paper, α = 0.5 works well across OPT/BLOOM/GLM/Llama. α = 0.85 is sometimes better for models with especially extreme outliers.

That’s it. One scalar per linear layer, computed from 128 calibration samples. No Hessian. Cheap.

4. What the math looks like on paper

Before SmoothQuant, activation channel j has values roughly in [-50, 50] (outlier channel). Per-tensor INT8 quant with scale s_act = 50/127. Bulk of activations at [-2, 2] maps to just 10 codepoints.

After smoothing with s_j = 25:

  • Activations become X'_j = X_j / 25, so range [-2, 2] for the outlier channel too. Now per-tensor INT8 quant has scale ~2/127, and bulk activations use the full range.

  • Weights become W'_j = 25 · W_j. Weights in that row grow, but since per-channel weight scales are already common practice, this is absorbed cheaply.

Activation quant error drops from catastrophic to small. Weight quant error stays small. Net effect: W8A8 works.

5. Where SmoothQuant sits in the family diagram

From 02_outlier_problem.md:

Family A: Isolate      → LLM.int8()
Family B: Migrate      → SmoothQuant, AWQ, QServe
Family C: Rotate       → QuaRot, SpinQuant, DuQuant

SmoothQuant is the archetype of Family B. AWQ is its cousin: same math (matmul scaling invariance), different goal (protect salient channels for W4A16 vs. flatten activations for W8A8).

6. Where SmoothQuant is used in 2026

As-is

Straight W8A8 SmoothQuant is still shipped for hardware where INT8 tensor cores are the right target and FP8 isn’t available or doesn’t help. Examples:

  • Older T4/V100 fleets (INT8 tensor cores, no FP8).

  • Ampere A100/A30/A40 (INT8 tensor cores, FP8 only from Hopper onward).

  • CPU inference paths where INT8 GEMMs (AMX on Sapphire Rapids, oneDNN) beat fp16.

On Hopper+ hardware, FP8 (see 07_fp8_family.md) is usually preferred over SmoothQuant W8A8 because:

  • FP8 handles outliers more gracefully (log-scale range).

  • FP8 doesn’t require calibration for scaling factor.

  • FP8 tensor cores have equal throughput to INT8 on H100.

As a foundation

The scaling migration idea is embedded in nearly every subsequent activation-quant method:

  • QServe (QoQ): SmoothAttention (SmoothQuant applied to KV cache).

  • AWQ: same invariance, different objective.

  • OmniQuant: makes s learnable via gradient descent.

  • Any “activation-aware weight quant” method: descendants.

7. Bit-level and kernel-level realities

SmoothQuant produces:

  • W' ∈ INT8 per-channel symmetric (one fp16 scale per output row).

  • X' ∈ INT8 per-token symmetric or per-tensor (one fp16 scale per token or per tensor).

  • Fused scaling factors into the preceding layer’s output. The migration X → X/s is mathematically absorbed into the previous layer’s weights (multiply the previous layer’s output-column-j weights by 1/s_j). Zero runtime cost.

The matmul itself becomes a straight INT8 × INT8 → INT32 tensor-core operation, followed by a dequant Y = scale_x · scale_w · Y_int32. This is what Ampere/Hopper INT8 tensor cores were designed for.

8. Quality expectations

Model

fp16 PPL (WT2)

Naive W8A8 PPL

SmoothQuant W8A8 PPL

OPT-6.7B

10.9

12,000+ (broken)

10.9–11.0

Llama-2-7B

5.47

42

5.5

Llama-3-8B

6.14

~30–80 (varies by layer)

6.20–6.30

Llama-3-70B

3.86

Similar breakage

3.87–3.90

The “naive W8A8 PPL” numbers are what you get if you skip SmoothQuant and just per-tensor quantize activations. Model dies.

Qualitatively: SmoothQuant at α=0.5 is nearly lossless for most Llama-class models at W8A8. If PPL is off by >0.2, you likely have a specific outlier layer (usually down_proj) that needs α=0.7–0.85.

9. Where SmoothQuant breaks down: W4A4

At 4-bit activations, SmoothQuant is no longer enough. Reason:

  • INT8 activations have 256 codepoints — you can afford some residual outlier magnitude after smoothing.

  • INT4 activations have 16 codepoints. Any residual outlier eats you.

SmoothQuant’s migration is bounded by how much difficulty you can dump into weights before those start clipping. That budget runs out below W8A8.

The answer, post-2024, is rotation methods — they don’t just migrate one channel’s difficulty to another; they spread the difficulty across all channels via an orthogonal transform. Covered in 14_projects.md (rotation family lightning tour); the papers to read are QuaRot (<phone_number_or_numberic_id_or_random_id_114>), SpinQuant (<phone_number_or_numberic_id_or_random_id_115>), and DuQuant (<phone_number_or_numberic_id_or_random_id_116>). These are the W4A4 SOTA in 2026.

10. How to run SmoothQuant in 2026

Most likely you’ll never run vanilla SmoothQuant — you’ll run it inside llm-compressor:

from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
from llmcompressor.modifiers.quantization import GPTQModifier

recipe = [
    SmoothQuantModifier(smoothing_strength=0.8),
    GPTQModifier(scheme="W8A8", targets="Linear", ignore=["lm_head"]),
]
# oneshot(...) as before

The recipe applies SmoothQuant scale migration first, then GPTQ error-compensating rounding on the migrated weights. This is what production W8A8 pipelines actually do.

Alternative: TensorRT-LLM’s own PTQ calibration flow implements SmoothQuant as a plugin. If you’re targeting TRT-LLM (Phase 7), you use that path.

11. study one-liner

SmoothQuant exploits the matmul scaling invariance X · W = (X/s) · (sW) to migrate quantization difficulty from activations (heavy-tailed, hard) to weights (bell-shaped, easy). It divides each activation input channel by a per-channel scalar s_j = max(|X_j|)^α / max(|W_j|)^(1-α) derived from calibration statistics, multiplies the matching weight rows by s_j, and folds the scaling into the previous layer for zero runtime cost. With α ≈ 0.5, this unlocks W8A8 INT8 tensor-core inference on Llama-class models at near-fp16 quality — which was impossible with naive per-tensor activation quant.

Next: 07_fp8_family.md — in 2026, if you’re doing activation quant on Hopper+ hardware, you’re mostly doing FP8, not INT8. Here’s why.