Rung 4 — The Quantization Bake-Off

Aligned phase: Phase 5 (quantization & model compression) Ship by: end of M10 Effort: ~3 weeks, ~30 hrs. Compute cost: $10–30 in rented H100 hours. Signal: high. This is the first rung a hiring manager reads and says “this person can prescribe deployments.” It is also the first rung r/LocalLLaMA will genuinely engage with.

The artifact spec

One 7–8B instruction-tuned model. Four production-relevant PTQ formats. Two tables. One prescription.

Reference build: 06_quantization/13_bake_off_project.md. That file has the copy-pasteable commands. This file tells you how to turn the raw output into a portfolio artifact.

Model: meta-llama/Llama-3.1-8B-Instruct or Qwen/Qwen2.5-7B-Instruct. Pick one, stick with it. No hedging, no “we tried both.”

Formats — exactly these four, no more, no fewer:

Format

Category

Bits/w (honest)

Why included

GPTQ W4A16 (g128, act-order)

Hessian-based weight-only

~4.25

The Marlin path on Ampere; the industry default

AWQ W4A16 (g128)

Activation-aware weight-only

~4.25

The alternative to GPTQ; often wins on Qwen

FP8_DYNAMIC (e4m3)

Weight+activation, Hopper-native

8.00

The 2026 production default for H100/H200

GGUF Q4_K_M (with imatrix)

CPU/edge/heterogeneous

~4.85 (see below)

The local/on-prem/mixed-hardware answer

Stretch (do at least one): GGUF IQ4_XS (~4.25 bpw, non-uniform codebook — direct comparison against Q4_K_M), or NVFP4 (~4.5 bpw with metadata) if you have B200 hours.

The two-table template — steal this exactly

Quality table

| Format          | WT2 PPL | ARC-C | MMLU | GSM8K | KL vs bf16 (mean/p95) | Bits/w |
|-----------------|---------|-------|------|-------|-----------------------|--------|
| bf16 (baseline) | 6.14    | 82.9  | 68.4 | 84.2  | — / —                 | 16.00  |
| FP8_DYNAMIC     | 6.16    | 82.8  | 68.2 | 84.0  | 0.002 / 0.008         | 8.00   |
| GPTQ W4-g128    | 6.28    | 82.1  | 67.5 | 82.5  | 0.012 / 0.041         | 4.25   |
| AWQ  W4-g128    | 6.31    | 82.0  | 67.3 | 82.1  | 0.014 / 0.048         | 4.25   |
| GGUF Q4_K_M     | 6.26    | 81.9  | 67.2 | 82.3  | 0.013 / 0.045         | 4.85   |

Speed table

| Format       | Decode B=1 tok/s | Decode B=32 tok/s | TTFT p95 (2k prompt) | ITL p95 | VRAM   |
|--------------|------------------|-------------------|----------------------|---------|--------|
| bf16         | 190              | 3200              | 210 ms               | 32 ms   | 16.8GB |
| FP8_DYNAMIC  | 330              | 5600              | 120 ms               | 20 ms   | 9.2 GB |
| GPTQ W4-g128 | 480              | 4100              | 240 ms               | 15 ms   | 5.4 GB |
| AWQ  W4-g128 | 475              | 4000              | 245 ms               | 15 ms   | 5.4 GB |
| GGUF Q4_K_M  | (llama.cpp)      | n/a               | (llama.cpp)          | —       | 5.5 GB |

These are illustrative numbers. Publish your own, on your own hardware, with pinned SHAs.

Honest reporting — the section that earns trust

r/LocalLLaMA readers can smell padding at 100 paces. These are the non-negotiables:

  1. Bits/w must be honest. Q4_K_M is ~4.85 bpw because K-quants carry per-super-block scales and mins. Do not round to “4.0” the way marketing decks do. Cite ikawrakow’s original PRs to llama.cpp for the derivation.

  2. NVFP4 is 4.5 bpw once you count the E4M3 block scale metadata, not 4.0. Ditto MXFP4 (4.25 bpw with E8M0 shared scale over blocks of 32). If you include either, spell out the accounting.

  3. KL-divergence to bf16 is the money metric. Perplexity moves ~2% between good and bad quants; KL moves 10×. Mean AND p95 — the tail is where quant damage hides.

  4. Report failures. If AWQ tanks GSM8K on your model, publish it. Do not silently drop the task. Reviewers who catch a missing task assume you’re hiding worse.

  5. Pin every version. transformers, vllm, llm-compressor, llama.cpp SHA. A table entry that can’t be re-run is a fake.

  6. No “quant beats bf16” claims. If your Q4_K_M shows +0.3 on MMLU, that’s eval noise, not compression magic. Run three seeds or don’t claim it.

The Marlin gap analysis — the follow-up post that lifts you above the crowd

After the four-format table ships, do one more experiment and either append it to the post or ship it as a sequel:

Write a naive W4A16 dequant+GEMM in Triton. Benchmark it against Marlin on the same GPTQ weights. Publish the gap chart.

The chart is: X-axis = batch size 1, 2, 4, 8, 16, 32, 64; Y-axis = tok/s. Two lines: your Triton kernel, Marlin. Expected shape:

  • Batch 1: your kernel is 30–50% of Marlin. Both are bandwidth-bound; the delta is in your dequant path being naive (not fused with the MMA, not double-buffered).

  • Batch 16+: your kernel is 10–25% of Marlin. This is where Marlin’s tensor-core pipelining, its interleaved weight layout, and its async global-loads compound.

The write-up is what matters, not the number. For each region of the curve, explain the gap with an Nsight Compute trace:

  • DRAM throughput (are you saturating bandwidth?)

  • Tensor core utilization

  • Shared memory bank conflicts on the dequant path

  • Instruction mix (are you spending too many cycles on shifts/masks vs. MMAs?)

This is what separates “did the benchmark” from “understands why quant kernels are hard.” The people who read Marlin’s source code and can explain the interleaved weight layout are hired to write the next one. That’s you after this exercise.

Reference: read 06_quantization/11_marlin_machete.md for the Marlin trick set. Read the Marlin repo README end-to-end before you start — it is a masterclass in Ampere optimization.

Repo structure

quant-bakeoff-llama31-8b/
├── README.md                # The blog post. Hero table at top.
├── scripts/
│   ├── quantize_gptq.py     # from llm-compressor
│   ├── quantize_awq.py
│   ├── quantize_fp8.py
│   ├── quantize_gguf.sh     # llama.cpp convert + imatrix + quantize
│   ├── run_lm_eval.sh
│   ├── run_kl_div.py
│   └── run_vllm_bench.sh
├── kernels/
│   └── w4a16_naive.py       # your Triton W4A16 for the gap analysis
├── bench/
│   ├── marlin_vs_ours.py
│   └── plot_gap.py
├── results/
│   ├── quality_table.csv
│   ├── speed_table.csv
│   ├── marlin_gap.png
│   └── ncu_traces/
│       ├── ours_b1.ncu-rep
│       └── ours_b16.ncu-rep
├── ATTRIBUTIONS.md          # ikawrakow, Tri Dao, Elias Frantar (GPTQ), Ji Lin (AWQ), Neural Magic (llm-compressor)
└── VERSIONS.txt             # every SHA, every pip freeze

Where to post

  1. GitHub: github.com/<you>/quant-bakeoff-llama31-8b. Pin the repo.

  2. Personal blog: cross-post the README. Title: “Llama-3.1-8B across 4 quant formats: quality × speed, with the Marlin gap explained.” No emoji, no clickbait. The title is the abstract.

  3. r/LocalLLaMA: submit on a Sunday afternoon IST / morning ET — that’s peak US engagement without the Monday firehose. Title of the post is the same as the blog title. Lead with the quality table image, put the link in the top comment as usual.

  4. HuggingFace Hub: upload the four quantized model artifacts as your own repos (<you>/Llama-3.1-8B-Instruct-GPTQ-W4A16 etc.). Cross-link them in the model cards to your write-up. This is where the “cited by a model card” success signal materializes — people who search for GPTQ Llama-3.1 land on your card, find your write-up.

  5. X/Twitter: thread with the two tables as images + Marlin-gap plot. Tag @vllm_project @lmsysorg @NeuralMagic. Do not tag Tri Dao unless you have the FA numbers to justify it.

  6. GPU MODE Discord #quantization: share once. If someone asks a question, answer it well.

  7. LinkedIn: post the write-up. First LinkedIn post if you skipped rung 3’s. Short: “Ran four quant formats through evals + benches. Tables + Marlin gap analysis. Link.”

Success signals (measure at 2 weeks)

  • Cited by at least one HF Hub model card (yours or someone else’s who reuses your quant recipe). This is the highest-signal outcome.

  • 3+ upvotes on r/LocalLLaMA with substantive comments. (Their vote distribution is bimodal; 20+ is a “hit,” 3+ with real replies is a legitimate contribution.)

  • 1 comment/star from a Neural Magic, vLLM, or llama.cpp contributor. DM Michael Goin (Neural Magic → RH) on X if the Marlin analysis is tight; his engagement is a real endorsement.

  • Referenceable URL for studies. Six months later you can paste this link and stop explaining your quant experience.

If ≥3 of 4 hit, ship rung 5. If only 1–2 hit, the write-up quality is the gap — not the numbers. Revise the prose before ditching the project.

What signals it sends

  • “I can operate at least four quantization toolchains without breaking them.”

  • “I understand why Q4_K_M is 4.85 bpw and not 4” — the honest-bits detail catches every reviewer’s attention.

  • “I ran lm-eval-harness on multiple variants without conflating tasks.”

  • “I can write a Triton W4A16 kernel and profile it against a hand-tuned CUDA reference.”

  • “I have prescription judgment, not just benchmark hands.” This is the phrase to plant in the write-up’s conclusion.

The Zoho angle (natural hook, don’t force it)

The bake-off’s closing section is a prescription table by hardware and batch. Add one paragraph:

“For on-prem SMB deployments with 2×L40S serving ≤32 concurrent users on a CRM-copilot workload — the scenario I see most at Zoho — the answer is GPTQ W4A16 with prefix caching enabled. Here’s why: [numbers].”

That paragraph is what turns the artifact from “student exercise” into “practitioner document.” Do not omit it. Do not sanitize the Zoho reference — the domain specificity is the credibility. But do not leak any customer detail; the paragraph is about the shape of the workload, not any specific tenant.

Past examples to study

  • r/LocalLLaMA weekly quant threads 2024–25 — search for ikawrakow and Nexesenex. The community reference for honest quant reporting. Read the top-voted comparison posts from Q2 2025 backwards.

  • Neural Magic / vLLM llm-compressor examples repo — https://github.com/vllm-project/llm-compressor/tree/main/examples. Their quantization recipes are the canonical llm-compressor usage. Cite them.

  • Marlin repo — https://github.com/IST-DASLab/marlin. Read marlin_cuda_kernel.cu alongside the paper. This is the reference for your gap-analysis section.

  • HuggingFace TheBloke model cards (historical) — the pattern for quant-card metadata even though the account is dormant. bartowski cards are the current live pattern.

  • ikawrakow’s original llama.cpp PRs for k-quants — the PR descriptions themselves are the derivation. Cite them by URL.

Common mistakes

  1. Skipping the imatrix step for GGUF. Q4_K_M without imatrix is measurably worse. If you’re comparing to production-quality Q4_K_M downloads, you need imatrix or the comparison is unfair to GGUF.

  2. Using default calibration for GPTQ/AWQ. 128 samples of open_platypus is the floor. For agentic use cases, calibrate on 128 samples of your target distribution (tool-call traces, CRM extractions) and note it. Calibration set choice is 30% of the quality delta.

  3. Benchmarking FP8 on Ampere. FP8 needs Hopper (H100/H200/L40S/RTX Ada) for tensor-core FP8; on Ampere it’s emulated and the speed numbers are meaningless. Either rent an H100 hour or drop FP8 entirely and note it explicitly.

  4. Batch-1 only measurements. Decode B=1 favors W4A16; B=32 favors FP8. Publishing only B=1 is dishonest — the entire prescription depends on batch regime. Both tables, always.

  5. KL-div computed on the wrong distribution. Compute logits on WT2 test, not train. Compute the top-k truncated KL if full-vocab KL is too memory-heavy; state the k.

  6. Marlin gap without Nsight traces. A speed gap without a profiled explanation is a shrug. The traces are 80% of the credibility for that section.

  7. Waiting for perfection. Ship at 4 formats + 4 tasks + KL. Do not delay to add SmoothQuant, W8A8, EXL2 — those are follow-up posts.

Success criteria (checklist before publishing)

  • Four formats quantized, weights on HF Hub with model cards

  • Quality table complete with 4 tasks + KL mean + KL p95

  • Speed table complete with B=1 and B=32, TTFT p95 and ITL p95

  • Bits-per-weight column is honest (Q4_K_M = 4.85, not 4.0)

  • Marlin gap analysis with at least 2 Nsight traces

  • Every command in the repo runs from a clean env in ≤1 hour

  • Zoho-flavored prescription paragraph in the conclusion

  • ATTRIBUTIONS.md credits ikawrakow, Frantar, Lin, Neural Magic

  • VERSIONS.txt with every pip freeze + llama.cpp SHA

If ≥7 of 9 check, ship. If ≥1 of the success signals hits within 2 weeks, you have a durable portfolio piece.

Next step

On the Monday after you publish this, you open your mini-engine repo (rung 5) and start integrating one of your bake-off formats — GPTQ W4A16 via compressed-tensors — as the engine’s first supported quantization path. The bake-off proved you understand formats; the engine proves you can serve them. Rung 4 feeds rung 5; that is the entire ladder.

The ladder is the CV. Every rung is public. Every rung compounds.