13 — THE Bake-Off Project: One 8B Model, Four Formats, Two Tables

This is the flagship portfolio artifact of Phase 5. Not because the numbers are novel — they aren’t — but because doing it end-to-end, with honest evals and reproducible commands, is what r/LocalLLaMA and the vLLM/SGLang contributor crowd actually respect. The reason you’ll get hired is because someone can read this post and immediately trust your judgement on their deployment.

Deliverable in one sentence

Take one 7–8B instruction-tuned model. Produce GPTQ W4A16 + AWQ W4A16 + FP8_DYNAMIC + GGUF Q4_K_M variants. Publish a speed table (batch 1, batch 32) and a quality table (WT2 PPL + 3 lm-eval tasks + KL-divergence to fp16), with every command copy-pasteable. Total effort: 4–6 evenings if you’re set up; ~2 weekends if you’re not.

Why exactly these four formats

Each represents a category, and together they span the entire commercially-relevant PTQ space:

Format

Category

Where it wins

GPTQ W4A16

Hessian-based weight-only

Bandwidth-bound decode on Ampere (Marlin)

AWQ W4A16

Activation-aware weight-only

Same regime, sometimes better at odd bit widths

FP8_DYNAMIC

Weight+activation (Hopper native)

Compute-bound prefill and high-batch serving

GGUF Q4_K_M

CPU/edge/heterogeneous

Local, offline, mixed GPU+CPU, on-prem SMB

Ship this table and you can prescribe formats for any customer scenario. That’s the whole point of Phase 5.

Model choice

Recommended: meta-llama/Llama-3.1-8B-Instruct or Qwen/Qwen2.5-7B-Instruct.

Reasons:

  • Both are well-studied — you’ll have reference numbers to sanity-check against.

  • Both are permissively licensed for research write-ups.

  • 7–8B is the sweet spot: small enough to iterate on a single 3090/4090/L40S, large enough that quant differences matter.

  • Instruction-tuned matters: base models don’t stress the chat-quality axis.

Do not use: Phi-3 (unusual architecture, quant tools misbehave); a raw base model (no MT-Bench signal); a 70B (too slow to iterate on).

The environment (pin these)

python == 3.11
torch == 2.5.1
transformers == 4.46.x
vllm == 0.7.x
llm-compressor == 0.4.x       # Neural Magic / vLLM, replaces AutoGPTQ + AutoAWQ + AutoFP8
lm-eval == 0.4.5
llama.cpp @ <commit SHA>      # pin a specific SHA, ecosystem moves fast

llm-compressor is the current recommended tool because it produces compressed-tensors format which vLLM loads natively, and it consolidates GPTQ/AWQ/FP8/SmoothQuant recipes in one framework. Repo: github.com/vllm-project/llm-compressor. AutoGPTQ is functionally deprecated; AutoAWQ is minimally maintained.

Hardware target for this bake-off: one H100 or one A100 80GB rented for $2–3/hr for a few hours. Or your local 3090/4090 24GB (Q4_K_M/GPTQ/AWQ all fit; FP8 needs H100/L40S/RTX 4090 Ada for native tensor-core FP8 — you can run FP8 emulated on 3090 for correctness testing but speed numbers won’t be meaningful).

Step 1 — Baseline (bf16)

# Load the fp16/bf16 model and run every eval you'll compare against.
# Do this ONCE, save the numbers.

MODEL_ID=meta-llama/Llama-3.1-8B-Instruct

lm_eval --model hf \
  --model_args pretrained=$MODEL_ID,dtype=bfloat16 \
  --tasks wikitext,arc_challenge,mmlu,gsm8k \
  --num_fewshot 0 \
  --batch_size 8 \
  --output_path ./eval-baseline-bf16.json

Speed baseline via vLLM:

vllm serve $MODEL_ID --dtype bfloat16 --max-model-len 4096
# in another shell:
vllm bench serve \
  --model $MODEL_ID \
  --dataset-name sharegpt \
  --num-prompts 200 \
  --request-rate inf

Report from this: TTFT p50/p95, ITL p50/p95, decode tok/s at B=1 and B=32. Screenshot the numbers.

Step 2 — GPTQ W4A16

# quantize_gptq.py
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

recipe = GPTQModifier(
    targets="Linear",
    scheme="W4A16",           # 4-bit weights, 16-bit activations
    ignore=["lm_head"],       # never quantize lm_head
    actorder="weight",        # act_order=True, the 2024 refinement (see 04_gptq)
    dampening_frac=0.01,
)

oneshot(
    model=model,
    tokenizer=tokenizer,
    dataset="open_platypus",  # or your domain calibration set
    recipe=recipe,
    max_seq_length=2048,
    num_calibration_samples=128,
    output_dir="./llama-3.1-8b-gptq-w4a16",
)

Wall time on one H100: ~15–20 minutes. Half of that is loading the model.

Step 3 — AWQ W4A16

# quantize_awq.py — same skeleton, different modifier
from llmcompressor.modifiers.quantization import AWQModifier

recipe = AWQModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=["lm_head"],
    bits=4,
    group_size=128,
    symmetric=False,          # asymmetric = zero-point; AWQ convention
    # search over ~20 alphas in [0, 1], picks per-layer
)

Same 128 calibration samples. Wall time ~10–15 minutes (no Hessian is faster than GPTQ’s).

Step 4 — FP8_DYNAMIC

# quantize_fp8.py — no calibration needed
from llmcompressor.modifiers.quantization import QuantizationModifier

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",     # dynamic per-token activation scale, static per-channel weight scale
    ignore=["lm_head"],
)

oneshot(model=model, recipe=recipe, output_dir="./llama-3.1-8b-fp8-dynamic")

Wall time: ~2 minutes. No forward pass through calibration data needed — dynamic activation scaling computes scales at inference time. This is FP8’s superpower: nearly zero conversion cost, nearly zero quality loss.

Step 5 — GGUF Q4_K_M

Convert via llama.cpp:

# 1. Convert HF to GGUF fp16 first
python llama.cpp/convert_hf_to_gguf.py $MODEL_ID_PATH --outfile llama-3.1-8b-f16.gguf

# 2. Compute imatrix (for i-quants; optional but recommended for Q4_K_M too)
./llama.cpp/build/bin/llama-imatrix \
  -m llama-3.1-8b-f16.gguf \
  -f wiki.train.raw \
  -o llama-3.1-8b.imatrix \
  --chunks 100

# 3. Quantize with imatrix
./llama.cpp/build/bin/llama-quantize \
  --imatrix llama-3.1-8b.imatrix \
  llama-3.1-8b-f16.gguf \
  llama-3.1-8b-q4_k_m.gguf \
  Q4_K_M

Wall time: ~10 minutes total. wiki.train.raw is the WikiText-2-raw training file; download from HuggingFace wikitext.

Bonus: also produce IQ4_XS (llama-quantize ... IQ4_XS) — it uses the same imatrix but a non-uniform 4-bit codebook, often edges out Q4_K_M at slightly lower bpw. Compare both in your table.

Step 6 — Quality evals

Run lm-evaluation-harness on each variant. GGUF via llama-server + --model hf won’t work directly; either:

  • Option A (HF-side quants): point lm-eval at each ./llama-3.1-8b-<format> directory.

  • Option B (GGUF): use llama-perplexity for PPL and llama-server + curl for task evals, OR convert back to HF format if the loader supports it (llama.cpp Python bindings + lm-eval local-completions model type).

for VARIANT in gptq-w4a16 awq-w4a16 fp8-dynamic ; do
  lm_eval --model hf \
    --model_args pretrained=./llama-3.1-8b-$VARIANT \
    --tasks wikitext,arc_challenge,mmlu,gsm8k \
    --num_fewshot 0 \
    --batch_size 8 \
    --output_path ./eval-$VARIANT.json
done

# GGUF via llama.cpp
./llama.cpp/build/bin/llama-perplexity \
  -m llama-3.1-8b-q4_k_m.gguf \
  -f wiki.test.raw \
  -c 2048

Step 7 — KL-divergence to fp16

The metric that catches what PPL misses. Script:

# kl_divergence.py
import torch, torch.nn.functional as F
from transformers import AutoModelForCausalLM, AutoTokenizer

def collect_logits(model_path, texts, device="cuda"):
    model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype="auto", device_map=device)
    tok = AutoTokenizer.from_pretrained(model_path)
    all_logits = []
    for t in texts:
        ids = tok(t, return_tensors="pt", truncation=True, max_length=2048).input_ids.to(device)
        with torch.no_grad():
            all_logits.append(model(ids).logits.float().cpu())
    return all_logits

fp16 = collect_logits(BASELINE, texts)
quant = collect_logits(QUANT, texts)

kls = []
for a, b in zip(fp16, quant):
    p = F.log_softmax(a, dim=-1)      # log p
    q = F.log_softmax(b, dim=-1)      # log q
    # KL(p || q) = sum p * (log p - log q)
    kl = (p.exp() * (p - q)).sum(-1).mean().item()
    kls.append(kl)

print(f"mean KL = {sum(kls)/len(kls):.5f}")

Run against WT2 test set (~4k samples). Report mean and p95 KL. This single number is the most convincing quality evidence you can publish.

Step 8 — Speed via vLLM

For each quant that vLLM loads natively (GPTQ, AWQ, FP8 — GGUF is llama.cpp only):

vllm serve ./llama-3.1-8b-<variant> \
  --quantization compressed-tensors \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.9

# benchmark
vllm bench serve \
  --model ./llama-3.1-8b-<variant> \
  --dataset-name sharegpt \
  --num-prompts 500 \
  --request-rate inf \
  --save-result

Report tokens/sec at B=1 (single sequential request) and at B=32 (concurrent). Also VRAM footprint (nvidia-smi after weights load, before requests).

For GGUF via llama.cpp:

./llama.cpp/build/bin/llama-bench \
  -m llama-3.1-8b-q4_k_m.gguf \
  -p 512 -n 128 -b 1 -b 32

Step 9 — The two tables (steal these templates)

Quality — Llama-3.1-8B-Instruct

| Format          | WT2 PPL | ARC-C | MMLU | GSM8K | KL vs bf16 | Bits/w |
|-----------------|---------|-------|------|-------|------------|--------|
| bf16 (baseline) | 6.14    | 82.9  | 68.4 | 84.2  | —          | 16.00  |
| FP8_DYNAMIC     | 6.16    | 82.8  | 68.2 | 84.0  | 0.002      | 8.00   |
| GPTQ W4-g128    | 6.28    | 82.1  | 67.5 | 82.5  | 0.012      | ~4.25  |
| AWQ W4-g128     | 6.31    | 82.0  | 67.3 | 82.1  | 0.014      | ~4.25  |
| GGUF Q4_K_M     | 6.26    | 81.9  | 67.2 | 82.3  | 0.013      | ~4.85  |
| GGUF IQ4_XS     | 6.29    | 81.8  | 67.1 | 82.0  | 0.014      | ~4.25  |

Speed — H100 SXM, vLLM 0.7.x / llama.cpp @ SHA

| Format          | Decode B=1 tok/s | Decode B=32 tok/s | Prefill 2k tok/s | VRAM  |
|-----------------|------------------|-------------------|------------------|-------|
| bf16            | 190              | 3,200             | 12,500           | 16.8 GB |
| FP8_DYNAMIC     | 330              | 5,600             | 24,000           | 9.2 GB  |
| GPTQ W4-g128    | 480              | 4,100             | 11,800           | 5.4 GB  |
| AWQ W4-g128     | 475              | 4,000             | 11,700           | 5.4 GB  |
| GGUF Q4_K_M     | (llama.cpp)      | —                 | —                | 5.5 GB  |

These numbers are illustrative; your job is to produce your own and explain the delta. In particular:

  • Why does FP8 beat W4A16 at B=32 (compute-bound crossover — go recompute the roofline math).

  • Why does W4A16 win at B=1 (bandwidth-bound — 4× less HBM traffic).

  • Why is bf16 prefill within ~50% of FP8 (both compute-bound, FP8 is 2× the tensor-core throughput minus overhead).

Step 10 — The write-up

Publish it. r/LocalLLaMA + your own blog + LinkedIn. Structure:

  1. The question. “Which 4-bit format should you actually deploy?”

  2. The setup. Model, hardware, versions, calibration data.

  3. The two tables above.

  4. The three surprises. Every serious bake-off reveals three counterintuitive findings — GSM8K delta between AWQ and GPTQ, or Q4_K_M beating GPTQ on MMLU, or FP8 losing at B=1. Highlight those.

  5. The prescription. Given the tables, when do you pick which. Frame by hardware and batch.

  6. The Zoho angle (optional but powerful): “for an on-prem SMB customer with 2× L40S serving <32 concurrent users, W4A16 GPTQ is the answer — here’s the math.”

A version of this post exists for every open model and every hardware family; almost no version is written well. Yours can be.

Extension: the Marlin gap analysis (file 14 in depth)

Once the four-format bake-off is out, do the follow-up: write a naive W4A16 dequant+GEMM Triton kernel, benchmark it against Marlin on the same GPTQ weights, produce a per-batch-size gap chart, and explain the gap using Nsight Compute traces. That project is described in 14_projects.md; treat it as the sequel post.

Anti-patterns that will kill your credibility

  • Reporting only one format’s numbers. The comparison is the entire value.

  • Skipping KL-divergence. Marks you as a PPL-only tourist.

  • Not pinning versions. Numbers are worthless if unreproducible.

  • Cherry-picking one favorable task. Include GSM8K even if your quant does badly on it. Especially then.

  • “My quant beats bf16 on task X.” No, it doesn’t — you have a noisy eval. Report multiple seeds or don’t.

  • Publishing without inspecting the outputs by hand. Generate 20 chat completions from each quant on the same prompts. Read them. If FP8 sounds different from bf16, PPL isn’t enough.

The Zoho translation

This bake-off is the internal document for every “which quant should we ship?” meeting you’ll have for the next three years. Do it once for your primary target model on your primary target hardware, then re-run it whenever either changes. Your customers don’t care about your GPTQ paper knowledge; they care that you’ve held four formats side-by-side and can prescribe one with numbers.

Homework

  1. Do steps 1–5 for one model. Save all files.

  2. Run all evals in step 6, plus KL-div step 7.

  3. Do the vLLM speed benches step 8.

  4. Fill in the two tables with your numbers.

  5. Write the post. Cross-post to r/LocalLLaMA on a Sunday afternoon (peak engagement) and to your own site.

  6. Bookmark the resulting URL. This is the artifact you point recruiters at.