14 — Phase 5 Projects Portfolio

The bake-off (file 13) is the flagship. This file is the supporting cast: smaller projects that build the foundational intuitions, plus the sequel-post “naive W4A16 Triton vs Marlin” analysis that is the single most technically-impressive artifact you can produce in Phase 5.

Order them roughly by dependency: 1–2 are prerequisites for reading the papers; 3–4 are prerequisites for reading the kernels; 5 is the flagship; 6 is the sequel; 7–8 are reference-post material; 9 is Phase 6 seed.

Project 1 — Affine quantization in numpy + activation histograms

Time: 1 evening. Purpose: internalize what quantization is, at the level of one tensor.

Write ~50 lines of numpy that implements:

def quantize(x, bits=4, group_size=128, symmetric=False):
    # returns (q_int, scales, zero_points) at chosen granularity
    ...

def dequantize(q_int, scales, zero_points):
    # returns approximate x
    ...

Then for each of {per-tensor, per-channel, per-group=128, per-block-2D=(128,128)} and each of {symmetric, asymmetric} and each of {int8, int4}:

  1. Draw 1000 samples from a bell-shaped N(0, 1) distribution.

  2. Draw 1000 samples from a heavy-tailed N(0, 1) + 0.001 * N(0, 100) distribution (models activation outliers).

  3. Quantize → dequantize → measure MSE and max absolute error.

  4. Plot histograms of the quantization error.

Expected finding: heavy-tailed distributions destroy per-tensor int4 (error blows up 10–100×), per-channel is a huge help, per-group is a modest additional help. Symmetric vs asymmetric matters more for heavy-tailed. This is the intuition behind literally every method in files 04–08, and having it in your fingertips means you’ll never confuse the granularity ladder again.

Bonus: repeat with a real activation tensor pulled from a hook on Llama-3-8B’s down_proj input. Watch the outlier channels light up.

Project 2 — Activation-inspection script (from file 02, formalized)

Time: 1 evening. Purpose: reproduce the outlier phenomenon from LLM.int8() and Massive Activations on any HF model.

Write a ~100-line PyTorch script that:

  1. Loads any HF causal LM (make it a CLI arg).

  2. Registers forward hooks on all nn.Linear modules that record input activations.

  3. Runs a small calibration set (a few chat prompts).

  4. For each layer, computes and dumps: per-channel max, per-channel std, per-token max, kurtosis, ratio of max-to-median.

  5. Plots the top-K “outlier channels” per layer as a heatmap over depth.

Expected finding: you’ll see the massive-activation channels (Sun et al. <phone_number_or_numberic_id_or_random_id_147>) pop out visually in specific layers, specific channels. You’ll see MLP down_proj inputs are worse than qkv inputs. This script becomes your first debugging tool whenever a quant misbehaves: run it, look for a layer with kurtosis >100, that’s your problem.

Reference: github.com/locuslab/massive-activations for their version of this.

Project 3 — GGUF header parser

Time: 1 evening. Purpose: demystify llama.cpp’s on-disk format so you can debug any GGUF file bit-by-bit.

Write a Python script that reads a GGUF file without importing gguf. It should print:

  1. Magic bytes (GGUF), version, tensor count, metadata KV count.

  2. All metadata key/value pairs (model type, tokenizer, RoPE config, etc.).

  3. For each tensor: name, shape, dtype (with the ggml type enum decoded to human name like Q4_K / Q6_K / F32), offset, size in bytes.

  4. Verify the bit-budget arithmetic: for a Q4_K_M model, iterate all tensors, sum bits = shape × bits/weight (using the table in 10_gguf_k_quants.md), compare to file size on disk. It should match within padding.

Spec reference: github.com/ggml-org/ggml/blob/master/docs/gguf.md (or the current llama.cpp docs equivalent).

Why this matters: GGUF-illiteracy is what separates people who use GGUF from people who understand it. This project fixes that in one evening. It also positions you to write the bartowski-quality quant tables that r/LocalLLaMA rewards.

Project 4 — Rotation-family reading & summary post

Time: 2–3 evenings (reading). Purpose: understand the current SOTA path for aggressive activation quant (W4A4), which the seed roadmap covered only in passing.

Read in order:

  1. QuaRot — arxiv <phone_number_or_numberic_id_or_random_id_148>, github.com/spcl/QuaRot. The seminal paper: apply a random Hadamard rotation to the residual stream. Since attention and MLP are invariant to orthogonal rotations of the hidden dim (up to careful matmul reordering), you can “rotate away” activation outliers without changing model outputs. Read carefully: which matrices absorb the rotation, why Hadamard specifically, what “online Hadamard” means for the ones that can’t be absorbed offline.

  2. SpinQuant — arxiv <phone_number_or_numberic_id_or_random_id_149>, github.com/facebookresearch/SpinQuant. Same idea, but LEARN the rotation via Cayley SGD on the Stiefel manifold instead of using random Hadamard. +45% gap-to-fp16 closure on Llama-3-8B W4A4 vs QuaRot.

  3. DuQuant — arxiv <phone_number_or_numberic_id_or_random_id_150>, github.com/Hsu1023/DuQuant. Dual rotation + zigzag permutation, specifically targets massive-activation spreading.

  4. KurTail — arxiv <phone_number_or_numberic_id_or_random_id_151>. Kurtosis-based rotation objective; cheaper than SpinQuant, better than QuaRot.

  5. ButterflyQuant — arxiv <phone_number_or_numberic_id_or_random_id_152>. Learnable butterfly transforms replace fixed Hadamard, O(n log n).

  6. PrefixQuant — arxiv <phone_number_or_numberic_id_or_random_id_153>, github.com/ChenMnZ/PrefixQuant. Complementary: isolates TOKEN-wise outliers by prefixing them to KV cache; +3 avg over SpinQuant on W4A4KV4.

Write a 1500-word summary post: “Rotation Methods for LLM Quantization, 2024–2025 in Review.” Table with method + arxiv + key trick + reported PPL gap on Llama-3-8B W4A4 vs fp16. This post will do exceptionally well on r/LocalLLaMA and establishes you as someone who tracks the frontier.

Why this matters commercially: as B200/RTX 5090 hardware rolls out, W4A4 (or NVFP4) via rotation methods will become the default for compute-bound serving. Anyone who understands this stack in <phone_number_or_numberic_id_or_random_id_160> is early.

Project 5 — THE bake-off (file 13)

See 13_bake_off_project.md. The flagship. 4–6 evenings. Non-negotiable.

Project 6 — Naive W4A16 Triton kernel vs Marlin: the gap analysis

Time: 1–2 weekends. Purpose: earn the right to talk about kernel-level quant performance.

This is the sequel to the bake-off, and the project that demonstrates you can move down the stack from “quant format user” to “quant kernel author.” It is one of the highest-signal artifacts you can produce in Phase 5 — because writing a Triton kernel that is slower than Marlin, then explaining exactly why, is more impressive than either doing neither or claiming a win against Marlin (nobody believes that).

The project:

  1. Take a GPTQ-quantized Llama-3.1-8B checkpoint from the bake-off. Load its qweight (int32-packed int4), scales (fp16, per-group), qzeros, g_idx on GPU.

  2. Write a Triton kernel w4a16_naive_matmul(x_fp16, qweight, scales, qzeros, g_idx) -> y_fp16 that:

    • Loads a tile of packed weights from HBM.

    • Unpacks and dequantizes to fp16 in registers/shared memory (do NOT materialize the full dequantized weight in HBM).

    • Does an fp16 tile matmul using Triton’s tl.dot.

    • Handles the group-size boundaries correctly.

  3. Verify numerical correctness against a reference dequant → fp16 matmul implementation across shapes: M ∈ {1, 4, 16, 64, 256}, K, N ∈ {4096, 8192, 11008, 14336} (the actual Llama shapes).

  4. Benchmark against Marlin (via vllm.model_executor.layers.quantization.marlin or the standalone Marlin package) on the same shapes and hardware. Report tokens-equivalent throughput per M.

  5. Profile both kernels with Nsight Compute at M=1 and M=32:

    • HBM bandwidth achieved (% of peak).

    • Tensor-core utilization (% of peak).

    • Shared-memory bank conflicts.

    • Register pressure.

    • Stall reasons.

  6. Write a post: “Why My Triton W4A16 Kernel Is 2–4× Slower Than Marlin, In Numbers.”

What you’ll find (spoilers, to save you the surprise):

  • At M=1, your Triton kernel and Marlin are both close to HBM-bandwidth-bound. You’ll be within ~30–50% of Marlin. Not embarrassing.

  • At M=16–32, Marlin pulls away 3–5×. Reasons:

    • Marlin uses cp.async for global→shared prefetching, hand-coded double-buffered pipeline.

    • Marlin’s weight layout is optimized for bank-conflict-free unpacking.

    • Marlin writes the epilogue with warp specialization, Triton does not.

    • Marlin has a hand-tuned occupancy that Triton’s auto-tuner can rarely match.

  • At M=1024+, both kernels shift compute-bound; Marlin’s advantage narrows but persists.

Write down every profile finding, screenshot Nsight, publish. This is the artifact that demonstrates you understand Phase 2–5 together, and it’s rare enough that it will attract attention.

Reference:

  • Marlin repo: github.com/IST-DASLab/marlin (READ THE README, it’s a masterclass).

  • Machete: read the vLLM PRs (vllm-project/vllm search for “machete”) for the Hopper successor’s design choices.

  • Aspirational Triton reference: arxiv <phone_number_or_numberic_id_or_random_id_154> (IBM SplitK W4A16 in Triton) shows how far skilled authors have pushed Triton for this shape.

Project 7 — KV cache quant sanity test (small)

Time: 1 evening. Purpose: internalize KV quant’s memory savings and quality tradeoff.

  1. Serve Llama-3.1-8B-Instruct on vLLM with a long-context prompt (say, RULER-style needle at 32k).

  2. Measure VRAM used at 3 settings: --kv-cache-dtype auto, --kv-cache-dtype fp8_e5m2, --kv-cache-dtype fp8_e4m3.

  3. Measure the accuracy on the needle test at each setting (score = did the model recall the injected fact?).

  4. Report: (a) VRAM saved by fp8 KV (~50%), (b) accuracy delta (should be ~0), (c) at what context length does VRAM become the binding constraint.

Write up as a 500-word supporting-cast post. Value: shows any customer with long-context workloads that fp8 KV is a free lunch.

Project 8 — 2:4 structured sparsity awareness demo

Time: 1 evening. Purpose: know what you’re not doing.

Compression-beyond-quant: NVIDIA Ampere+ tensor cores support 2:4 structured sparsity (every group of 4 weights has 2 zeros). Combined with W4A16, this gives a further ~2× speedup on compatible shapes.

  1. Read the Sparse-Marlin section of the Marlin README.

  2. Read llm-compressor’s SparseGPTModifier docs.

  3. Take one linear layer from your bake-off model, apply SparseGPT 2:4, quantize with GPTQ W4, verify: (a) 50% of weights are zero in each row-4 group, (b) inference still works, (c) quality delta is modest.

  4. Note: full production 2:4 recipes need training-aware or careful PTQ to preserve quality; awareness is enough for Phase 5.

Report: is 2:4 + W4A16 a good combo for your target hardware? (Ampere: yes; Hopper: mostly; Blackwell: superseded by native FP4.)

Project 9 — QLoRA fine-tune bridge (into Phase 6)

Time: 1 evening. Purpose: bridge PTQ (Phase 5) into training-time quant (Phase 6).

  1. Read the QLoRA paper (arxiv <phone_number_or_numberic_id_or_random_id_155>). Understand NF4 (NormalFloat 4-bit, information-theoretically optimal for zero-mean normal distributions), double quantization (quantize the fp32 quantization constants themselves), paged optimizers.

  2. Fine-tune Llama-3.1-8B with bitsandbytes NF4 + LoRA on any small instruction dataset (Dolly-15k, OpenAssistant sample). ~1 GPU-hour on a 3090.

  3. Merge the LoRA weights back, produce a full-precision checkpoint, then run the full bake-off (from file 13) on it. Compare to the base model’s bake-off numbers.

Value: you now have hands-on experience with NF4 (the datatype), LoRA/QLoRA (the parameter-efficient training method), and the merge → quantize → serve pipeline. This is the ROI story for on-prem customer fine-tunes at Zoho: “we fine-tuned an 8B on your data with QLoRA on a single GPU, then quantized for serving.”

Awareness-level projects (do not build, just know)

Distillation

How the small-model ecosystem is actually made. Every Phi/Qwen-1.5B/Llama-3.2-1B was distilled from a bigger teacher. Read the DistilBERT and MiniLM papers for the mechanics; read the Phi-3-mini technical report for the modern data-curriculum approach. Not a Phase 5 build, but essential context: distillation + quantization is the real “how do I run a smart model on limited hardware” answer, not either alone.

Low-rank / MLA-style KV compression

DeepSeek-V2 arxiv <phone_number_or_numberic_id_or_random_id_156>: Multi-head Latent Attention compresses KV into a joint low-rank latent (kv_lora_rank ≈ 512), then re-projects at attention time. This is not quantization but is a KV-cache-compression technique that layers on top of KV quant. Read the paper; you’ll come back to it in Phase 6.

QAT (Quantization-Aware Training)

At PTQ scale (7B–70B), PTQ + rotation methods are close enough to QAT that few actually run QAT anymore. But awareness: LLM-QAT (arxiv <phone_number_or_numberic_id_or_random_id_157>) and its descendants, and BitNet a4.8 (native 4-bit activation training). QAT scaling law: arxiv <phone_number_or_numberic_id_or_random_id_161>. Read one QAT paper end-to-end; that’s your Phase 5 QAT quota.

Product quantization / codebook quantization

AQLM (Additive Quantization for LMs) and QuIP# push weights below 2 bits via learned codebooks. Impressive but slow; use awareness-level.

The exercise answer key (from file 01)

From 01_quant_theory.md, the five practice exercises. Answers so you can grade yourself:

  1. Store 4096 fp32 weights as INT4 per-tensor asymmetric. Bits = 4×4096 = 16384 bits for the codes + 32 bits for scale + 32 bits for zero-point = 16448 bits = 2056 bytes. Bits/weight ≈ 4.016. Correct.

  2. Same weights per-group asymmetric g=128. Groups = 4096/128 = 32. Overhead = 32 × (16 fp16 scale + 16 fp16 zero) = 1024 bits = 128 bytes. Codes = 2048 bytes. Total = 2176 bytes. Bits/weight ≈ 4.25.

  3. Same weights MXFP4. Groups = 4096/32 = 128. Overhead = 128 × 8 bits (E8M0) = 1024 bits = 128 bytes. Codes = 2048 bytes. Total = 2176 bytes. Bits/weight ≈ 4.25 (matches per-group INT4 g=128; coincidence of accounting).

  4. Same weights NVFP4. Groups = 4096/16 = 256. Overhead = 256 × 8 bits (FP8-E4M3 block scale) + 32 bits (FP32 tensor scale) = 2080 bits = 260 bytes. Codes = 2048 bytes. Total = 2308 bytes. Bits/weight ≈ 4.51.

  5. Per-token dynamic scaling of an activation tensor of shape [T, C]. Scales stored per-row = T fp16 scales = 2T bytes. Codes = T×C bytes for INT8. Overhead in bits/element = 16/C ≈ tiny for C=4096. This is why per-token activation quant is essentially free at inference.

Project sequencing summary

Week

Project

1

Projects 1, 2 (numpy quant + activation inspection)

2

Project 3 (GGUF parser) + start reading rotation family (Project 4)

3–4

Project 5 (the bake-off) — flagship

5–6

Project 6 (Triton W4A16 vs Marlin) — sequel

7

Projects 7, 8 (KV quant + 2:4 sparsity)

8

Project 4 write-up (rotation family) + Project 9 (QLoRA bridge into Phase 6)

Eight weeks, six write-ups, one flagship. This is what Phase 5 looks like when you actually do it.

Homework

  1. Sequence these projects into your calendar. Block the weekends for Projects 5 and 6.

  2. Set up one public repo phase-5-quantization where every project lands as a subdirectory with its own README and reproducible commands.

  3. Every project’s README ends with a link to the accompanying blog post (r/LocalLLaMA or your site). Zero orphan projects.

  4. At the end of Phase 5, tag the repo v1.0-phase-5-complete — that tag is the artifact you point recruiters at, and the entry ticket to sustained OSS contribution in Phase 7.