06 — Phase 1 Projects (What You Ship)

Three projects. Each has: goal, deliverable, acceptance criteria, honest time budget, portfolio value. You do not exit Phase 1 without projects 1 and 2 shipped publicly. Project 3 is stretch and highly recommended.

Every project follows the discipline: predict → measure → explain the gap → publish.

Project 1 — From-Scratch Inference Matching HF (the Phase 1 capstone)

Goal: Write, from scratch, an inference-only implementation of one modern small LLM (Llama-3.2-1B / Qwen2.5-1.5B / SmolLM2-1.7B — pick one) in PyTorch that loads real HuggingFace weights and produces token-identical greedy output to transformers.AutoModelForCausalLM.

Repo layout (target):

from-scratch-inference/
├── README.md              # what, how to run, results table
├── model.py               # ModelConfig, RMSNorm, RoPE, GQA Attention, SwiGLU, TransformerBlock, LlamaModel
├── kv_cache.py            # pre-allocated KVCache class
├── tokenizer.py           # thin wrapper over `tokenizers` lib with a from-scratch BPE toy for demo
├── sampling.py            # LogitsProcessor chain: greedy, temperature, top-k, top-p, min-p, rep-penalty
├── load_hf.py             # safetensors → your model, with the weight-name map
├── generate.py            # CLI: python generate.py --model llama-3.2-1b --prompt "..." --max-new-tokens 200
├── tests/
│   ├── test_token_identical.py   # greedy match vs HF on 20 prompts
│   ├── test_logits_diff.py       # per-token max|logits_diff| < 1e-3
│   └── test_kv_correctness.py    # cached generation matches naive full-context
└── benchmarks/
    ├── kv_speedup.py             # naive vs cached decode; plot
    └── decode_tok_per_sec.py     # predicted vs measured tok/s

Acceptance criteria (all must pass):

  1. Token-identical greedy vs HF for 20 diverse prompts, at least 200 tokens each. (argmax matching, not just perplexity closeness.)

  2. Working KV cache with a benchmark plot showing near‑constant per‑token latency (cached) vs quadratic wall‑clock (naive), measured over context lengths 128 → 2048.

  3. GQA correctly implemented — verify n_kv heads not n_q. Print the KV cache size at 4k context and match it to the arithmetic in 03_transformer_arithmetic.md.

  4. RoPE correctly implemented including the HF Llama half‑half convention (dim i pairs with dim i + d_head/2, not i+1). This is the #1 cause of “almost right” outputs.

  5. Sampling ladder implemented as a LogitsProcessor chain (temperature → top‑k → top‑p → min‑p → repetition penalty). Demonstrate temperature sweep.

  6. README with the arithmetic: parameter count derivation, KV bytes/token, predicted vs measured decode tok/s on your hardware. Predicted number MUST be committed before the measured number.

Time budget: 40–60 hours. Be honest about which HF weight names map to which of your modules; that alone is 3–5 hours of frustration and is worth documenting for others.

Portfolio value: very high. This is the artefact that proves Phase 1 exit. Every subsequent portfolio piece assumes this is real. Post it publicly with the numbers. If you can’t ship this you cannot ship anything downstream.

Stretch: add Qwen3-VL-tiny awareness (skip vision but confirm the text pathway); add YaRN‑style RoPE scaling and generate at 2× original context.

Project 2 — The Transformer Arithmetic Notebook (the study artefact)

Goal: A single Jupyter notebook that, given any HF model config, computes parameter count, FLOPs per token, KV cache bytes, memory budget, and predicted batch‑1 decode tok/s. Verified against ≥5 real models.

Notebook layout:

transformer-arithmetic.ipynb
1. ModelConfig dataclass (auto-loads from HF config.json)
2. param_count(cfg) → returns int; verified against HF model.num_parameters() for:
   - Llama-3.2-1B
   - Llama-3.1-8B
   - Llama-3.1-70B (from config only, don't need to download weights)
   - Qwen2.5-7B
   - DeepSeek-V3 (bonus: MoE + MLA — shows off you handle both)
3. flops_per_token(cfg, seq_len) → (dense_flops, attn_flops); table across models
4. kv_cache_bytes(cfg, seq_len, batch, dtype) → returns bytes; the horror table for Llama-3-70B at 128k
5. max_batch_size(cfg, gpu_mem_gb, seq_len, weight_bits, kv_bits)
6. predicted_decode_tps(cfg, hbm_bw_gbs, weight_bits) → compare to measured on your hardware
7. Roofline plot (matplotlib): ridge = peak_flops/hbm_bw ; scatter your naive matmul, torch matmul, cuBLAS from your Phase 0 work; kernels from Phase 2–3 will be added later
8. Master table: {model} × {precision} × {gpu} → predicted_tps vs measured_tps vs gap%

Acceptance criteria:

  1. Parameter count within ±1% for 5 real models (accounting for tied vs untied embeddings correctly).

  2. KV cache table for Llama‑3‑70B at 128k in fp16, fp8, int4 — numbers match 03_transformer_arithmetic.md exactly.

  3. Predicted vs measured decode tok/s for at least three (model, precision, GPU) triples. Predictions must land within ±25% (this is the calibration target).

  4. One post‑mortem paragraph per triple explaining any gap larger than 25%.

Time budget: 15–25 hours.

Portfolio value: very high. This is the “can you do napkin math on silicon” study artefact. Put it in your README top‑level. It is worth more than any certificate.

Project 3 — The Paper‑Read Notebook (proof of taste)

Goal: A markdown file with your one‑paragraph five‑question summary of every paper in 05_paper_canon_phase1.md, plus one original observation per paper (something you noticed that isn’t in the abstract).

Format (per paper):

## Paper Title (Author, Year) — arxiv link

**Problem**: [1-2 sentences]
**Prior state of the art**: [1 sentence]
**The one trick**: [1-2 sentences, the crux]
**Numbers that changed**: [tabular or terse]
**The buried tradeoff**: [what the paper doesn't shout about]
**My original observation**: [1-3 sentences — what connects this to another paper, an implementation detail you noticed, an obvious follow-up]

Acceptance criteria: 10 papers minimum (Section canon), the DeepSeek V2 + V3 pair counted separately.

Time budget: 15–25 hours. This is paper reading time, not writing time. The writing forces the reading to be honest.

Portfolio value: medium‑high. It won’t get you hired on its own, but combined with Project 1 and 2, it establishes you as “reads primary sources and thinks about them”, which is a much rarer signal than most people assume.

Public Shipping Checklist (for each project)

Before you call anything “done”:

  • Repo has a clean README with pip install -r requirements.txt (or uv sync) and a one‑line run command.

  • Results table with exact hardware, exact commit hash, exact commands to reproduce.

  • At least one plot (matplotlib PNG committed to the repo).

  • A predicted_vs_measured section, even if the predictions were wrong — especially then.

  • A short blog post (Substack / personal site / dev.to / X thread) linking the repo, walking through one insight. The post is the receipt.

  • Cross‑posted to r/LocalLLaMA (Project 1) and/or the GPU MODE Discord #showcase channel for feedback.

The Meta‑Deliverable: The Lab Notebook

Across all three projects, maintain a running LAB.md file with dated entries:

### 2026-04-14  Attempt 1 at RoPE
- **Hypothesis**: rotating dims (2i, 2i+1) will match HF.
- **Predicted**: token-identical output.
- **Measured**: outputs diverge after ~3 tokens.
- **Root cause**: HF Llama rotates halves (i, i + d_head/2), not adjacent pairs.
- **Fix**: rewrote apply_rotary_emb to use half-half convention. Match restored.
- **Lesson**: check the *specific* convention of the *specific* codebase before assuming the paper's math.

This is the file that turns a Phase 1 archive into a legible record of your growth. Show excerpts of it in the blog post from Project 1. study partners who see this file know you can be trusted with production systems.

Exit Check for Phase 1

You are done with Phase 1 (and ready for Phase 2 GPU work) when you can look at this checklist and every box is honestly ticked:

  • Project 1 shipped and publicly visible; token‑identical to HF greedy on 20 prompts.

  • Project 2 shipped; parameter count and KV cache formulas verified against ≥5 models; decode tok/s predicted within ±25% for ≥3 (model, precision, GPU) triples.

  • Project 3 shipped with ≥10 paper summaries.

  • You can whiteboard, from memory, in one sitting: the parameter breakdown of a transformer block, the KV cache formula, why decode is memory‑bound, the online softmax recurrence.

  • You have a blog post (or equivalent public writeup) walking through Project 1’s failures and lessons.

If yes to all: you are now in the top 5% of self‑taught inference engineers on the internet, and everything in Phases 2–7 has scaffolding to stand on. If no to any: don’t proceed. The exit criteria exist for a reason. The failure mode of this roadmap is people who “kind of” finished Phase 1 and then get destroyed by Phase 3.