02 — The Outlier Problem: Why Naive INT8 Falls Off a Cliff on LLMs¶

Thesis: Every quantization method invented since 2022 is an outlier-management strategy. If you don’t feel the outlier problem in your bones, none of the later methods will make sense.

1. The picture you must burn into your visual memory¶

Take a Llama-3 or Qwen2.5 8B model. Pick any middle transformer block. Run one forward pass on a real prompt. Now histogram the input activations to the down_proj (i.e., the output of SwiGLU).

You will see:

~99.9% of the values are within [-2, +2].
~0.1% are between 20 and 100.
A handful of activations in a handful of specific channels are >200 — sometimes >1000.

A per-tensor INT8 activation quant on this distribution sets s = max/127 ≈ 8, which means the 99.9% of tokens that live in [-2, +2] map to {-16, -8, 0, 8, 16, ...} — five codepoints. Your model dies. Perplexity jumps 10–100×. This is the problem.

2. LLM.int8() — the paper that named the problem¶

Dettmers, Lewis, Belkada, Zettlemoyer (2022) — LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. arxiv <phone_number_or_numberic_id_or_random_id_112>. Bundled into bitsandbytes.

The finding¶

Above ~6.7B parameters, LLMs develop emergent outlier features: specific channels of the hidden state whose magnitude is 6–10σ above the rest. These outliers are:

Systematic — the same channels are outliers across nearly all tokens.
Load-bearing — zeroing them destroys the model. They carry attention-sink-like information.
Concentrated — in Llama-class 7B models, ~6 channels out of 4096 contain them.

A histogram threshold of |x| > 6.0 catches nearly all of them.

The fix (LLM.int8() itself)¶

Mixed-precision decomposition of the matmul:

Vector-wise quantize the “normal” columns of X to INT8 and rows of W to INT8 → compute in INT8 tensor cores.
Detect outlier columns of X (any column with any |x| > 6.0).
Compute the outlier portion in fp16 (small sub-matrix, negligible cost).
Sum the two partial results.

Result: 8-bit inference at essentially zero PPL degradation, even for OPT-175B. But the fp16 side-path breaks GPU throughput because it forces a synchronization and a non-tensor-core kernel. LLM.int8() is correct but slow.

Why this paper matters historically¶

It was the diagnosis. Every method after this is a treatment.

SmoothQuant: eliminate outliers by migration → pure INT8, fast.
AWQ: protect the channels that see outliers → 4-bit weights only.
GPTQ: use error compensation to survive quant errors from outliers.
QuaRot/SpinQuant: rotate the space so no channel is an outlier.
LLM.int8() decomposition itself: MosaicQuant (<phone_number_or_numberic_id_or_random_id_113>, 2026) revives the split, done in modern fused kernels.

3. Massive Activations — the deeper, weirder story¶

Sun, Chen, Kolter, Liu (2024) — Massive Activations in Large Language Models. arxiv <phone_number_or_numberic_id_or_random_id_114>. Repo: github.com/locuslab/massive-activations.

A year and a half after LLM.int8(), a second paper looked closer and found something even weirder. Beyond the ~6σ outliers Dettmers found, there exist activations that are:

~100,000× larger than the median activation. Not 10×. Not 100×. Ten to the fifth.
Input-independent — the same tokens (typically <bos> or a delimiter) in the same layers produce the same magnitude regardless of prompt.
Load-bearing in a specific way: they act as implicit attention biases. If you zero them out at inference time, quality collapses. If you replace them with a constant learned bias, quality is preserved.

Why this changes the game¶

These aren’t distributional outliers you can rotate away — they are the model’s learned mechanism for attention sinks. Any quantization method that hopes to be fully 4-bit-activation has to either:

Preserve them explicitly (PrefixQuant, arxiv <phone_number_or_numberic_id_or_random_id_115>: prefix the outlier tokens to the KV cache and quantize everything else),
Rotate them away in a way that survives the residual stream (QuaRot / SpinQuant: Hadamard rotation on the residual stream is what makes W4A4 viable),
Or train the model to not have them (BitNet: teach the model from scratch to live at low precision — the outliers never form).

4. Why activations are harder to quantize than weights¶

A table to internalize:

	Weights	Activations
Distribution	Roughly bell, roughly zero-mean, low kurtosis	Heavy-tailed, high kurtosis, channel-outliers
Static or dynamic?	Static — known at load time, can spend hours computing scales	Dynamic — changes per token, must be quantized at runtime
Access pattern	Read once per matmul	Streaming, one token at a time in decode
Cost of getting it wrong	Model-wide, forever	Immediate, per-token, compounding

This is why weight-only quantization (GPTQ, AWQ) was solved first (2022–2023) and why activation-quantization to 4 bits is still an active research frontier in 2026.

5. The three families of outlier-management, in one diagram¶

                ┌─────────────────────────────────────────────┐
                │        The outlier problem in LLMs         │
                └───────────────────┬─────────────────────────┘
                                    │
        ┌───────────────────────────┼───────────────────────────┐
        ▼                           ▼                           ▼
  "Isolate them"              "Migrate them"             "Rotate them away"
                                                        
  LLM.int8() ('22)          SmoothQuant ('22)          QuaRot ('24)
  MosaicQuant ('26)         AWQ ('23)                  SpinQuant ('24)
  PrefixQuant ('24)         QServe/QoQ ('24)           DuQuant ('24)
  Dense+Sparse KVQuant ('24)                            KurTail ('25)
                                                        ButterflyQuant ('25)
                                                        TORQ ('25)

Each column represents a strategy, and every method fits into exactly one column (some — QServe, MosaicQuant — combine two).

Family A: Isolate¶

Keep outliers in high precision (fp16 or INT8) and quantize the rest aggressively. Works, but breaks kernel fusion.

Family B: Migrate¶

Rescale the network so activation outliers become weight outliers (weights are static → we have time to handle them). SmoothQuant is the archetype; AWQ generalizes it. Works well up to W8A8 and W4A16. Fails at W4A4 because migration can only move so much difficulty.

Family C: Rotate¶

Multiply activations by an orthogonal matrix (Hadamard) so the outlier’s magnitude gets spread across all channels. Mathematically invariant (the matmul result is unchanged if you multiply weights by the inverse rotation), computationally cheap (Hadamard is O(n log n)). This is the only family that has demonstrably reached W4A4 with near-fp16 quality — the state of the art in mid-2026.

Details in 04_gptq.md, 05_awq.md, 06_smoothquant.md, and the QuaRot/SpinQuant/DuQuant coverage in 14_projects.md (rotation family is deep enough for its own file if you want to expand later).

6. “But my activations look fine” — how to check your own model¶

Do this before you quantize anything. Ten lines of PyTorch:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B", torch_dtype=torch.bfloat16, device_map="cuda")
tok   = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B")

target_layer = model.model.layers[15].mlp.down_proj   # arbitrary middle layer
acts = {}

def hook(mod, inp, out):
    acts["x"] = inp[0].detach().float().cpu()

h = target_layer.register_forward_hook(hook)
model(**tok("The quick brown fox jumps over the lazy dog.", return_tensors="pt").to("cuda"))
h.remove()

x = acts["x"].flatten(0, -2)              # [tokens, channels]
print("shape:", x.shape)
print("abs max :", x.abs().max().item())
print("abs p50 :", x.abs().quantile(0.50).item())
print("abs p99 :", x.abs().quantile(0.99).item())
print("abs p9999:", x.abs().quantile(0.9999).item())
print("kurtosis :", ((x - x.mean())**4).mean() / (x.var()**2))

# Per-channel maxes reveal outlier channels
ch_max = x.abs().max(dim=0).values
top = torch.topk(ch_max, 8)
print("top-8 channels:", top.indices.tolist(), "| max mags:", top.values.tolist())

What you’ll see for a 7B+ model: p50/p9999 ratio of 20–100, per-channel top-8 magnitudes ~50–500, kurtosis in the thousands. This is the shape of the enemy.

Run this on your target model as your first act of any quantization project.

7. Where in the network the outliers live¶

Also from the 2024–2025 literature (Diagnosing FP4, arxiv <phone_number_or_numberic_id_or_random_id_117>; QAT scaling law, arxiv <phone_number_or_numberic_id_or_random_id_118>):

MLP down_proj input (post-SwiGLU) — the worst offender. This is where the massive activations live.
MLP up_proj / gate_proj input (post-RMSNorm) — medium-hard. Rotation-friendly.
Attention q_proj/k_proj/v_proj input (post-RMSNorm) — similar to gate/up.
Attention o_proj input (attention output) — mostly benign.
Early layers are often harder than middle layers under MXFP4 (surprising — the assumption used to be middle-layer worst).

When you inspect your model, focus first on down_proj inputs. If those look manageable, you can quantize the rest.

8. The study answer¶

When someone asks “why does INT8 activation quant hurt LLMs?”, the correct answer in one paragraph:

LLMs above ~6.7B parameters develop emergent outlier channels — a handful of channels whose activation magnitudes are 10–100× larger than the median. Because uniform affine quantization sets its scale from the max, one outlier channel forces the scale so high that the bulk 99% of activations map to just 3–5 codepoints, destroying resolution and thus perplexity. Beyond that, some activations are “massive activations” that are 10⁵× the median and act as implicit attention biases — they must be preserved. The entire modern quantization stack is a set of strategies for handling these outliers: LLM.int8() isolates them in an fp16 sidepath, SmoothQuant migrates their difficulty into weights, AWQ scales the channels that carry them, and QuaRot/SpinQuant rotate the space so no channel is an outlier at all.

That paragraph is a 30-second pitch. Say it out loud. Then keep going to 03_asymmetry_rule.md.