01 — Foundations Papers¶
Phase alignment: Months 1–3 (Phase 0/1). Read these while building your from-scratch inference GPT.
The purpose of this set is not to teach you the transformer — Karpathy does that better. The purpose is to teach you why the transformer looks like it does today, which is a story about inference economics as much as modeling quality. When you finish this set you should be able to answer: why GQA and not MHA? why RoPE and not learned positional embeddings? why did MLA replace GQA in the DeepSeek line?
1. Attention Is All You Need¶
arxiv:1706.03762 · Vaswani et al., 2017 · [EASY] · Prereqs: none
Key trick: Replace recurrence with parallelizable dot-product attention plus positional encodings, making sequence models trainable in one giant matmul instead of N sequential steps.
What to extract:
The exact Q/K/V/O projection shapes. Draw them.
Why √d_k scaling in softmax (numerical stability of the exp).
Multi-head is not “multi-attention” — it’s a reshape of one big projection. Prove this to yourself.
The FFN block: 2 linear layers with a 4x expansion. Every model since 2017 uses this ratio (until SwiGLU changed the constant, not the shape).
Encoder-decoder is a red herring for you. LLMs are decoder-only. Note the causal mask.
Note trap: the paper’s positional encoding (sinusoidal) is not what modern LLMs use. Read RoPE next to fix your model.
2. Language Models are Unsupervised Multitask Learners (GPT-2)¶
No arxiv — OpenAI tech report · Radford et al., 2019 · [EASY] · Prereqs: paper #1
Key trick: Show that a large enough decoder-only transformer trained on enough web text does zero-shot tasks purely by prompting, ending the pretraining+finetuning era.
What to extract:
Decoder-only stack. LayerNorm placement (pre-norm in GPT-2, not post-norm like the original). This one detail affects training stability at scale.
Byte-pair encoding tokenizer at 50k vocab.
Skim the results, focus on architecture.
3. Language Models are Few-Shot Learners (GPT-3)¶
arxiv:2005.14165 · Brown et al., 2020 · [EASY] (skim) · Prereqs: paper #2
Key trick: At 175B params, in-context learning emerges strongly enough to be the primary usage mode — scale is the algorithm.
What to extract:
The scaling table (Section 2.1). Memorize the shape: as params grow, layers grow slowly, hidden dim grows fast, heads grow with hidden dim.
Sparse attention variants (alternating dense/sparse) — obsolete now, but explains why later papers focus on making dense attention faster.
Compute budget: 3.14e23 FLOPs. Anchor for scaling-law reasoning.
Skim the eval results. You don’t care about SuperGLUE numbers; you care about the architecture and compute story.
4. Training Compute-Optimal Large Language Models (Chinchilla)¶
arxiv:2203.15556 · Hoffmann et al., 2022 · [MEDIUM] · Prereqs: paper #3
Key trick: For a fixed compute budget, optimal training uses ~20 tokens per parameter — GPT-3 and its peers were dramatically undertrained.
What to extract:
The 20-tokens-per-parameter rule. Not gospel (Llama-3 was trained on 15T tokens for 8B and 70B — wildly over-Chinchilla for inference reasons), but the reasoning is gospel.
Why post-Chinchilla, models are trained way past Chinchilla-optimal: inference cost dominates over training cost for widely deployed models. Smaller model + more training tokens = same quality but cheaper to run forever. This economic point is the reason you have a job to prepare for.
The three fitting methods (isoflop curves, parametric loss fit, direct fit) — skim.
Cost: the paper measures at ~1B–16B scale. Extrapolation to 100B+ was famously unstable.
5. RoFormer: Enhanced Transformer with Rotary Position Embedding (RoPE)¶
arxiv:2104.09864 · Su et al., 2021 · [MEDIUM] · Prereqs: paper #1
Key trick: Encode positions by rotating Q and K in 2D subspaces by an angle proportional to position, giving relative-position properties for free without adding parameters.
What to extract:
The rotation matrix per pair of dims. Derive on paper that <RoPE(q,m), RoPE(k,n)> depends only on m−n.
Why theta_i = 10000^(−2i/d) — the frequency schedule.
Implementation reality: it’s applied to Q and K after projection, in the head_dim space, in pairs. Look at Llama’s RoPE code (github.com/meta-llama/llama/blob/main/llama/model.py).
NTK-aware scaling and YaRN (arxiv:2309.00071) as follow-up reading: how to extend context beyond training length. You will need this in Phase 6 when 128k context is a requirement.
Implement it: in your from-scratch inference GPT (Phase 1), swap sinusoidal for RoPE. Verify against HF’s rotary implementation via matched outputs.
6. LLaMA: Open and Efficient Foundation Language Models (Llama-1)¶
arxiv:2302.13971 · Touvron et al., 2023 · [EASY] · Prereqs: papers #1–5
Key trick: Take GPT-3-style architecture, apply a handful of Chinchilla-aligned improvements (RMSNorm, SwiGLU, RoPE, no bias, longer training), release the weights, and change the field.
What to extract:
RMSNorm vs LayerNorm: no centering, no bias, one learnable scale per dim. Cheaper, works.
SwiGLU in the FFN: gate_proj(x) ⊙ silu(up_proj(x)) → down_proj. Note the FFN now has three linears, not two, so the hidden dim shrinks from 4x to ~2.67x to keep parameter count.
Pre-norm placement (norm before attention/FFN, residual outside).
No bias in linear layers.
Training details: cosine LR, AdamW, gradient clipping.
This paper is worth reading because every subsequent open model copies this recipe with small tweaks. Once you own Llama-1, Qwen/Mistral/Yi/DeepSeek-base architectures are ~90% the same.
7. Llama 2: Open Foundation and Fine-Tuned Chat Models¶
arxiv:2307.09288 · Touvron et al., 2023 · [EASY] (skim architecture section) · Prereqs: #6
Key trick: Same recipe as Llama-1 but with GQA at 34B/70B, longer context (4k), and a serious RLHF pipeline.
What to extract:
GQA arrives here in a flagship model. See paper #10 below.
Skim the RLHF section. Interesting but not your target skill — you’re the person who makes RLHF-trained models fast, not the person who does the RLHF.
8. The Llama 3 Herd of Models¶
arxiv:2407.21783 · Grattafiori et al., 2024 · [HARD] · Prereqs: #7
Key trick: Nothing individually novel — but the ~90-page report is the most detailed public description of a 400B-scale training run’s infrastructure ever written. Read it as an infra document.
What to extract: (mandatory sections marked ⭐)
⭐ Section 3 (Pre-Training) — the 4D parallelism (TP/PP/CP/DP) and the actual numbers on how much each cost. Failure rates: 419 unexpected interruptions in 54 days on 16K H100s.
⭐ Section 3.3 (Infrastructure) — storage, network, RDMA config, load-balancing, NCCL tuning knobs.
Section 3.4 (Training recipe) — read once.
Section 5 (Results) — skim.
Skip everything vision/multimodal on first pass.
Notice: inference-time considerations dictated architecture choices (GQA, tokenizer size, context length).
This paper is 90 pages. Budget a full weekend. Come back to it in Month 6 and again in Month 13.
9. Fast Transformer Decoding: One Write-Head Is All You Need (MQA)¶
arxiv:1911.02150 · Shazeer, 2019 · [EASY] (it’s a 6-page note) · Prereqs: #1
Key trick: Share one K head and one V head across all Q heads. KV cache shrinks by n_heads×, decoding gets memory-bandwidth-fast, quality drops slightly.
What to extract:
The KV cache is the bottleneck for long-context serving. Shazeer saw this in 2019.
Quality loss is real but small — and it goes away with GQA (below).
This paper is the ancestor of every modern KV-reduction technique: GQA, MLA, cross-layer attention, KV compression.
Do the KV math yourself for a Llama-3-8B config: MHA vs MQA vs GQA cache sizes at 128k context. Feel the difference.
Geoff Hinton once said the best papers are 6 pages. This is one.
10. GQA: Training Generalized Multi-Query Transformer Models¶
arxiv:2305.13245 · Ainslie et al., 2023 · [EASY] · Prereqs: #9
Key trick: MHA has n_heads KV, MQA has 1 KV — GQA has G KV where each shares across n_heads/G Q heads. G=8 for Llama-3 recovers ~all MHA quality with the KV-cache reduction of near-MQA.
What to extract:
The uptraining recipe: convert MHA → GQA by mean-pooling KV heads then continue pretraining for ~5% of original compute. Almost lossless.
Llama-3-8B uses 32 Q heads, 8 KV heads (G=8, ratio 4). Llama-3-70B same G=8, ratio 8. Compute your own KV memory math.
Now you can read the vLLM/SGLang KV-cache code and understand why the block dimension is (num_kv_heads, head_dim), not (num_heads, head_dim).
11. DeepSeek-V2: A Strong, Economical, and Efficient MoE (MLA)¶
arxiv:2405.04434 · DeepSeek-AI, 2024 · [HARD] · Prereqs: #10, some MoE awareness
Key trick: Multi-head Latent Attention — compress K and V through a shared low-rank latent (~d/16) that gets projected up per head. KV cache stores only the latent, so cache-per-token drops another ~4x below GQA, without the head-count ratio limitation.
What to extract:
The absorbed matmul trick: at inference, W_UK and W_UV can be absorbed into W_Q and W_O so the up-projection cost vanishes. This is the reason MLA is fast, not just small.
Decoupled RoPE: RoPE doesn’t commute with the low-rank compression, so a small separate rotary-carrying head is added. Understand why this is necessary — rotational position can’t be applied inside the compressed subspace.
The economics: DeepSeek uses this at 236B / 21B-active. The whole architecture is designed around inference-time cost.
Read alongside the DeepSeek-V3 paper (Phase 6) which uses the same MLA.
Cost: the paper’s math is dense. You will need two passes. Their blog + Sebastian Raschka’s writeup are the best companions.
The map you should be able to draw after this set¶
On a whiteboard, from memory:
Attention variants family tree: MHA (2017) → MQA (2019) → GQA (2023) → MLA (2024). At each node write: KV cache size in bytes for one token of Llama-3-8B-equivalent config.
Modern-stack cheat sheet: Pre-norm + RMSNorm + SwiGLU + RoPE + GQA + no-bias = every open model 2023–now. Deviations = interesting research.
The Chinchilla economic pivot: train small models past Chinchilla-optimal on lots of data → cheap serving → that’s why 7–8B models are the ecosystem’s center of gravity.
If you can’t draw the family tree — and put the numbers on it — you haven’t finished this set. Reread paper #9 through #11.
Optional additions (only if you’re ahead of schedule)¶
Cross-Layer Attention (arxiv:2405.12981) — further KV reduction by sharing across layers. Interesting but not yet in mainline engines.
YOCO: You Only Cache Once (arxiv:2405.05254) — self-decoder + cross-decoder, cache from one layer. Awareness only.
Massive Activations in LLMs (arxiv:2402.17762) — the empirical explanation of why quantization is hard. Read before Phase 5.