Phase 1 — Transformers Down to the Metal

Duration: Weeks 4–14 (starts in parallel with Phase 0) Budget on the 13-month viking timeline: ~8 weeks of focused work at 15–20 hrs/week. Prerequisite: You can code Python fluently. You’ve watched at least the first two Karpathy videos. You’ve drawn bf16’s bit layout from memory.

The Philosophy

You will build a GPT from scratch. Twice. The first time to learn how it thinks; the second time to make it fast.

Most engineers who “know transformers” know them at the level of model = AutoModel.from_pretrained(...). That is not what you’re building. You are building the level below: safetensors bytes → tokenizer → embedding → attention with a KV cache you allocated yourself → RoPE you rotated with your own math → GQA broadcast you implemented → sampler with logit processors you chained.

By the end of Phase 1, HuggingFace transformers will feel like a wrapper, not a black box. You will be able to point to the exact five lines in your own code that correspond to any model.generate() behavior. This is the level of ownership that Phase 2 (writing kernels for these ops) requires.

The killer artifact: an arithmetic notebook that lets you predict — before running anything — the parameter count, FLOPs/token, KV cache size, and memory-bound decode speed of any transformer from its config JSON. When a senior engineer at an onsite study says “we’re serving Llama-3-70B on 4×H100 with 128k context,” you should be doing the KV-cache math in your head before they finish the sentence.

The Six Subtopics

#

File

What you’ll own

1

01_karpathy_path.md

Zero to Hero → nanoGPT → llm.c → nanochat, with what to type and what to skip

2

02_from_scratch_inference.md

Round 2 build: safetensors format, BPE, RoPE, KV cache (naive → cached), GQA, sampling ladder — with code sketches

3

03_transformer_arithmetic.md

THE killer notebook: param count formula, FLOPs/token derivation, worked KV-cache example on Llama-3-70B at 128k (fp16 vs fp8)

4

04_cs336.md

Stanford CS336 Spring-2025 status, assignments, how to use it as your backbone curriculum

5

05_paper_canon_phase1.md

The 10 papers, one paragraph each on what to extract

6

06_projects.md

The concrete deliverables

Exit Criteria (all four)

You are done with Phase 1 when:

  1. HF-token-identical inference. Your from-scratch inference script generates greedy-decoded output that matches transformers.AutoModelForCausalLM.generate(do_sample=False) token-for-token for a real 1B–8B model (Llama-3.2-1B, Qwen2.5-1.5B, or SmolLM2-1.7B all work). This is a hard verification: any bug in RoPE, GQA broadcasting, or KV cache off-by-one shows up immediately.

  2. KV cache working. You’ve implemented naive re-run-attention decoding, then a KV cache, measured the wall-clock ratio, and can explain the O(n²) → O(n) crossover mathematically.

  3. Arithmetic notebook. You can, from a config JSON alone, compute: (a) parameter count within 1% of the actual checkpoint; (b) KV cache bytes per token; (c) predicted batch-1 decode tokens/sec on your GPU. All three verified against a real model.

  4. Papers read. You’ve read Attention Is All You Need, GPT-2/3, RoPE, Llama-3, Chinchilla, MQA, GQA, and DeepSeek-V2 (MLA). One paragraph of notes per paper explaining the problem, the one trick, and the tradeoff.

Reading Order Within Phase 1

  • Week 1 of the phase: Karpathy Zero-to-Hero videos 1–4 (micrograd → makemore → makemore MLP → wavenet). Type every line. Do NOT copy-paste.

  • Week 2: Karpathy video 5 (GPT / nanoGPT from scratch). Train a tiny model on TinyShakespeare on your 3090.

  • Week 3: Skim llm.c’s train_gpt2.c (~1000 LOC pure C). This is your bridge to Phase 2.

  • Weeks 3–5: Round 2 build (02_from_scratch_inference.md). This is the meat.

  • Weeks 4–8: Papers, in parallel with build (05_paper_canon_phase1.md).

  • Weeks 5–7: Arithmetic notebook (03_transformer_arithmetic.md).

  • Weeks 4–14 (background): CS336 lectures + assignments 1 and 2 (04_cs336.md).

Phase 1 ends when your inference matches HF token-for-token AND your notebook predicts KV cache size within 5% of measured. Do not skip either check.

The Meta-Habit for Phase 1

Extend the Phase 0 lab notebook to include architectural predictions:

Model:                  Llama-3.2-1B
Config predicted params: 1.235 B
Actual params:          1.24 B (from state_dict)
Gap:                    <1% — good.

KV cache per token:     130.56 KB (predicted, fp16)
KV cache measured:      130.56 KB (via allocated bytes)
Gap:                    0% — arithmetic is exact.

Batch-1 decode t/s:     predicted 42 t/s (on RTX 3090, 936 GB/s HBM, 1B fp16 = 2 GB weights)
Measured:               38 t/s
Gap:                    -10% — CPU launch overhead + some non-weight bandwidth.

This exact format — hypothesis, arithmetic, measurement, gap — is what separates senior inference engineers from junior ones. Do it every time.