12 — Evaluation Discipline: No Eval = Vandalism

Quantization work without evals is vandalism. A speedup claim without a quality table is a vibe. This file is the discipline layer that separates you from the LinkedIn crowd. Every compression claim ships with a quality table AND a speed table.

The four axes every eval must cover

  1. Language modeling loss — perplexity on a held-out corpus (WikiText-2, C4). Insensitive but universal, cheap to compute.

  2. Downstream tasks — zero/few-shot accuracy on a fixed suite (ARC-C, MMLU, HellaSwag, TruthfulQA, GSM8K). Sensitive to actual reasoning/knowledge.

  3. Distributional similarity — KL-divergence to the fp16 baseline on real distributions. The r/LocalLLaMA favorite; catches damage invisible to PPL.

  4. Behavioral spot-checks — chat quality (MT-Bench, IFEval), long-context (RULER, LongBench), your own domain. The tail failures live here.

Skip any of these and you’re guessing.

Perplexity on WikiText-2: the conventions

PPL is the standard opening ritual. There are hidden pitfalls:

Which dataset: wikitext-2-raw-v1 from HuggingFace. NOT wikitext-2-v1 (which has UNK tokens and is a different distribution). Always “raw”.

Stride and sliding window: the classic Sliding-window perplexity computes loss on the last stride tokens of each 2048-context window. Convention:

  • Context length: 2048 tokens (matches most 2023-era model contexts).

  • Stride: 512 tokens (75% overlap gives smoother estimates).

  • Sum negative log-likelihood over all target tokens; divide by total token count; exponentiate.

Which tokens count: only the final stride tokens of each window count for the mean NLL (not the whole window — tokens near position 0 have no context and get artificially high PPL). Getting this wrong is why different tools report different WT2 PPL for the same model.

Rule of thumb PPL bands (Llama-3-8B, WT2 raw-v1, 2048 ctx):

  • fp16: ~6.14

  • FP8_DYNAMIC: ~6.15–6.17 (essentially free)

  • GPTQ W4-g128 no act_order: ~6.35

  • GPTQ W4-g128 + act_order: ~6.25

  • AWQ W4-g128: ~6.30

  • GGUF Q4_K_M: ~6.25

  • GGUF IQ4_XS: ~6.30

  • GGUF Q3_K_M: ~6.60

  • W4A4 rotation (SpinQuant): ~6.65

These are the numbers you should have in your head. Report the delta, not the absolute.

Which tool:

  • llama-perplexity (llama.cpp binary) — the community reference for GGUF files.

  • lm-eval --task wikitext — for HF models (safetensors, GPTQ, AWQ, FP8).

  • These two produce SLIGHTLY different numbers (different tokenization padding, different window handling). Report which one you used; never mix.

Anti-pattern: “my quant loses 0.02 PPL vs fp16” without saying which tool, which stride, which subset. This is uninterpretable.

lm-evaluation-harness — the standard task battery

Repo: github.com/EleutherAI/lm-evaluation-harness. Current version as of <phone_number_or_numberic_id_or_random_id_158>: v0.4+. This is the reference tool for reproducible downstream evals. Every serious quantization paper reports its numbers through this harness.

Install and run:

pip install lm-eval
lm_eval --model hf \
        --model_args pretrained=./llama-3.1-8b-fp8-dynamic \
        --tasks arc_challenge,hellaswag,mmlu,truthfulqa_mc2,gsm8k \
        --num_fewshot 0 \
        --batch_size 8 \
        --output_path ./eval-fp8-dynamic.json

The standard “quant paper” 5-task battery:

Task

What it measures

Sensitivity

Notes

ARC-Challenge

Grade-school science reasoning

Medium

Zero-shot standard

HellaSwag

Commonsense continuation

Low

Least sensitive; sanity check

MMLU

57-task knowledge (5-shot)

High

The one everyone reports

TruthfulQA-mc2

Truthful vs plausible-sounding

Medium

Catches specific quant damage patterns

GSM8K

Grade-school math (8-shot CoT)

Very high

Quant damage is worst here — always include

Why GSM8K matters most: chain-of-thought reasoning is fragile. Losing 3–5 points on MMLU is bad; losing 15 points on GSM8K under W4A4 is common. If your quant only reports MMLU, be suspicious. GSM8K is where FP4 methods sink or swim.

Backend consistency: always run the fp16 baseline on the same lm-eval version, same GPU, same batch_size, same seed as the quantized runs. Cross-version comparisons are worthless.

KL-divergence to fp16: the honest metric

What r/LocalLLaMA has quietly settled on as the truth-teller for quant quality.

Definition: For each token position t on a held-out corpus, compute the softmax distribution p_fp16(·|context_t) from the fp16 baseline and p_quant(·|context_t) from the quantized model. Measure KL(p_fp16 || p_quant), averaged over positions.

Why it’s the honest metric:

  • PPL only sees the log-prob of the correct token. If your quant redistributes mass over incorrect tokens (still assigns the correct one a decent prob), PPL barely moves — but the model has become less like fp16 in every other way.

  • KL-div sees the full distribution shift. It catches: temperature-effective changes, entropy inflation, top-k pollution, all the ways a quant can “still work” while being different.

  • It correlates better with subjective chat quality than PPL does.

Standard implementation: run both fp16 and quant model over a fixed corpus (WT2 test set), collect logits at every position, compute KL per position, report mean and 95th percentile. Both llama.cpp (--kl-divergence) and independent scripts do this.

Rule of thumb KL bands (Llama-3-8B, WT2 test):

  • FP8_DYNAMIC: ~0.001–0.003 (negligible)

  • GPTQ W4-g128 + act_order: ~0.008–0.015

  • AWQ W4-g128: ~0.010–0.020

  • GGUF Q4_K_M: ~0.010–0.020

  • GGUF IQ3_M: ~0.03–0.05

  • W4A4 rotation: ~0.06–0.10 (visible)

  • Naive W4A4: 0.5+ (broken)

Any quant with KL > 0.1 on a normal corpus needs justification.

Long-context spot-checks

Quant damage compounds with context length. A quant that ties fp16 on WT2 (2048 ctx) may fall apart at 32k. This is where KV-cache quantization damage lives, and where naive W4A4 activation quant most often breaks.

RULER (arxiv 2404.06654): synthetic haystack + variant benchmarks up to 128k. Standard “needle in haystack” plus multi-key, multi-value, multi-hop variants. Report the accuracy curve as context length grows — fp16 should be flat, a bad quant will droop past 8k.

LongBench: real-world long-context tasks (single-doc QA, multi-doc QA, summarization, few-shot). More realistic than RULER, more expensive to run.

LongBench-v2 is the current standard as of <phone_number_or_numberic_id_or_random_id_159>. Use it if you have compute; RULER’s needle test if you’re cheap.

The rule: always report at least one long-context number. “My W4 quant matches fp16 on MMLU” is not enough if the model degrades at 32k — which is the context that matters for real usage.

Chat-quality spot-checks

More subjective, more expensive, but catches failures the above miss. Quant damage on instruction-tuned models often shows up as “model is dumber in a way PPL can’t see.”

MT-Bench: ~80 multi-turn prompts across 8 categories. Judged by GPT-4 (or Claude). Expensive per run (~$5-10 in judge calls). Report as a 0-10 mean.

IFEval: instruction-following on strict format tasks. Objective grading. Cheap. Very sensitive to quant damage — use as a first pass before MT-Bench.

AlpacaEval 2 / Arena-Hard-Auto: LLM-as-judge win rate against a reference. Report win-rate delta.

Rule: for any instruction-tuned model quant, run IFEval at minimum. If it drops >3 points, run MT-Bench before publishing. If MT-Bench drops >0.2, do not ship.

Reproducibility hygiene (non-negotiable)

Every table in your bake-off needs to be runnable by a stranger. That means:

  • Pin versions. transformers==X.Y.Z, lm-eval==0.4.N, vllm==0.M.N, llama.cpp commit SHA. Put them in the README.

  • Fix seeds. Sampling temperature 0 (greedy) for eval; explicit seeds anywhere randomness enters.

  • Lock the hardware statement. Which GPU, which CUDA driver, which nvidia-smi clock state.

  • Commit the calibration data. Not “128 random C4 samples” — the exact 128 samples, hashed.

  • Publish the commands. Copy-pasteable, not “the standard settings.”

  • Report percentiles for speed, not means. p50/p95/p99 tokens/sec on a fixed workload. Never a single number.

Benchmark hygiene sub-checklist for speed:

  • Warmup ≥ 5 iterations before measurement.

  • Lock GPU clocks: sudo nvidia-smi -lgc <base>,<base> to defeat thermal variance.

  • Use CUDA events for GPU timing, not Python time.time().

  • Report tokens/sec at batch=1 AND at least one higher batch (32 is standard).

  • Include the arrival pattern for serving benchmarks (Poisson λ, or explicit fixed rate).

The eval table template (steal this)

Here is the table shape every quant post should have:

### Quality — Llama-3.1-8B-Instruct (WT2 raw-v1, 2048 ctx, lm-eval v0.4.5)

| Format          | WT2 PPL | ARC-C | MMLU | GSM8K | KL vs fp16 | Notes |
|-----------------|---------|-------|------|-------|------------|-------|
| bf16 (baseline) | 6.14    | 82.9  | 68.4 | 84.2  | —          |       |
| FP8_DYNAMIC     | 6.16    | 82.8  | 68.2 | 84.0  | 0.002      |       |
| GPTQ W4-g128    | 6.28    | 82.1  | 67.5 | 82.5  | 0.012      | +act_order |
| AWQ W4-g128     | 6.31    | 82.0  | 67.3 | 82.1  | 0.014      |       |
| GGUF Q4_K_M     | 6.26    | 81.9  | 67.2 | 82.3  | 0.013      | llama.cpp |

### Speed — H100 SXM, vLLM 0.7.x (batch=1 decode, batch=32 concurrent)

| Format          | Decode B=1 tok/s | Decode B=32 tok/s | Prefill 2k tok/s | VRAM |
|-----------------|------------------|-------------------|------------------|------|
| bf16            | 190              | 3,200             | 12,500           | 16.8 GB |
| FP8_DYNAMIC     | 330              | 5,600             | 24,000           | 9.2 GB  |
| GPTQ W4-g128    | 480              | 4,100             | 11,800           | 5.4 GB  |
| AWQ W4-g128     | 475              | 4,000             | 11,700           | 5.4 GB  |
| GGUF Q4_K_M     | (llama.cpp only) | —                 | —                | 5.5 GB  |

Numbers above are illustrative-realistic. Your actual numbers will differ; that difference is the write-up.

The Zoho translation

On-prem customer deals are won and lost on evals like this. When your customer’s compliance team asks “how much quality did we lose by quantizing?”, the answer needs to be a single-page table, not a demo. Building the internal eval harness (a variant of this template, plus your customer’s domain-specific tasks) is a Phase 4.1/7.1 skill — do it early, it makes you the arbiter of every future model/quant choice at the company.

Anti-patterns

  • “Perplexity is fine” as the only claim. PPL misses too much. Always add MMLU/GSM8K + KL-div at minimum.

  • “vs fp16” without saying which fp16. Full precision loaded how? On which GPU? With which tokenizer? Reproducibility.

  • Cherry-picking one favorable task. Report the full battery or you’re lying.

  • Comparing across lm-eval versions. MMLU scoring subtly changed between 0.3.x and 0.4.x; a “quant improved MMLU by 0.5” is often just a version bump.

  • Reporting means for latency. Serving is a p99 game. Means hide the tail.

  • Skipping chat evals for instruction-tuned models. A 4-bit “Instruct” model can still write. Whether it follows instructions is a separate question.

The two-sentence study answer

Every compression claim ships with a quality table (PPL + 3-5 tasks via lm-eval-harness + KL-div-to-fp16) and a speed table (batch=1 and batch=32, p50/p95/p99), pinned to versions and hardware — anything less is vandalism. KL-divergence is the honest metric because it sees the full distribution shift; PPL only sees the correct-token log-prob and misses the ways a quant becomes different-but-still-plausible.

Homework

  1. Set up lm-evaluation-harness locally, run the 5-task battery on Llama-3.1-8B-Instruct bf16 as your baseline. Write down every version and hardware detail.

  2. Compute WT2 PPL for the same model with llama-perplexity (GGUF fp16) and lm-eval --task wikitext (HF fp16). Note the delta. Understand why they differ before you use either seriously.

  3. Write a 30-line KL-divergence script: two forward passes, softmax, torch.kl_div per position, mean over corpus. Test on the same model at fp16 vs fp8 — you should see near-zero KL.

  4. Bookmark: the current lm-eval version’s changelog, the RULER paper, the MT-Bench repo. These are your eval canon and will get referenced in every bake-off you publish.