05 — The Paper Canon for Phase 1

Read these papers, in this order, over Phase 1 (weeks 4–14). Each paper is dense; each reward is that half of what everyone else says about LLMs falls into place as a corollary of something you understood at the source. The habit: one paragraph in your notes per paper answering these five questions: (1) What problem? (2) What was the state of the art before? (3) What single trick? (4) What numbers changed? (5) What did it cost — what tradeoff did the paper bury?

Every paper below has been checked to still be canonical in 2026. Where a paper has been meaningfully superseded I’ve noted the current standard.

1. Attention Is All You Need — Vaswani et al., 2017

arXiv: https://arxiv.org/abs/<phone_number_or_numberic_id_or_random_id_15>

Extract: The transformer as an encoder‑decoder machine‑translation model. Even though you will never use this exact architecture (modern LLMs are decoder‑only, pre‑norm, with GQA/RoPE/SwiGLU/RMSNorm), every term in the original attention formula — the sqrt(d_k) scaling, the softmax rescaling, multi‑head splitting, positional encodings — is still load‑bearing in your Phase 1 code.

What to extract carefully:

  • The exact formula softmax(QK^T / sqrt(d_k)) · V.

  • Why sqrt(d_k) (variance analysis in Section 3.2.1).

  • The multi‑head split as “same total FLOPs, different geometry” (each head sees a lower‑dim subspace).

  • The parameter breakdown of the FFN (d → 4d → d) — you will replace this with SwiGLU, but the FLOP count survives.

What NOT to trust as modern: sinusoidal positional encoding (superseded by RoPE), layer norm placement (post‑norm loses to pre‑norm at scale), ReLU FFN (superseded by SwiGLU).

Companion: Jay Alammar’s “The Illustrated Transformer” (https://jalammar.github.io/illustrated-transformer/) is still the best visualisation. Read it, then the paper, then re‑read it.

2. Improving Language Understanding by Generative Pre‑Training (GPT‑1) & Language Models are Unsupervised Multitask Learners (GPT‑2) — Radford et al., 2018/2019

GPT‑2 paper: https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

Why: This is where the decoder‑only, causal‑mask, next‑token‑prediction stack you’re building becomes The Default. Skim GPT‑1 briefly; read GPT‑2 for architecture (Section 2), scaling behaviour hints, and its (understated) empirical claim that scaling + next‑token prediction = general capability.

Extract: (a) the architectural table (12/24/36/48 layers, hidden 768/1024/1280/1600); (b) the observation that byte‑pair encoding at byte level solves tokenization universality; (c) the vibe of “we just made it bigger and it worked” — which sets up Kaplan/Chinchilla.

3. Language Models are Few‑Shot Learners (GPT‑3) — Brown et al., 2020

arXiv: https://arxiv.org/abs/<phone_number_or_numberic_id_or_random_id_16>

Extract: The scaling table (175B params, 96 layers, 12288 hidden, 96 heads, 128 head‑dim) — memorise these numbers as reference points against which you’ll compare every subsequent open model. Also, the alternating dense/sparse attention pattern (a pre‑FlashAttention hack; nobody bothers now that FA exists).

Skip: the meta‑learning philosophy is dated. Read for numbers and architecture.

4. RoFormer: Enhanced Transformer with Rotary Position Embedding — Su et al., 2021

arXiv: https://arxiv.org/abs/<phone_number_or_numberic_id_or_random_id_17>

Extract: The positional encoding of every modern LLM. Understand:

  • What rotation does: pairing dims (2i, 2i+1) and rotating by angle m·θ_i (where θ_i = 10000^{−2i/d} in vanilla RoPE) encodes position m such that <RoPE(q, m), RoPE(k, n)> depends only on (m−n) — relative position “for free” inside the dot product.

  • Why it’s better than sinusoidal add: relative rather than absolute, works with the attention dot product without extra parameters, extrapolates (partially) to longer contexts.

  • The base θ matters: Llama‑3 uses θ = 500000 instead of the original 10000. Larger base → longer wavelengths on the low‑frequency dims → better long‑context behaviour. This is a free lever.

Companion papers on RoPE extensions (read these when tackling long context):

  • Position Interpolation (PI) — Chen et al., 2023 (https://arxiv.org/abs/<phone_number_or_numberic_id_or_random_id_18>): rescale positions to fit inside the trained range. Loses high‑frequency information; needs fine‑tuning.

  • NTK‑aware scaling — originated on Reddit (r/LocalLLaMA, u/bloc97), formalised later: don’t scale positions, scale the base θ. Free training‑less extension.

  • YaRN — Peng et al., ICLR 2024 (https://arxiv.org/abs/<phone_number_or_numberic_id_or_random_id_19>): piecewise frequency scaling + attention temperature. The current standard. Llama‑3.1’s rope_scaling (factor 8×, from 8k → 131k) is a YaRN‑style scheme. Repo: https://github.com/jquesnelle/yarn.

  • LongRoPE — Microsoft, 2024 (https://arxiv.org/abs/<phone_number_or_numberic_id_or_random_id_20>): search for per‑dim scaling factors via evolutionary search. Extends to 2M+ tokens. Awareness‑level.

Best 2025 explainer of the whole APE→RoPE→PI→NTK→YaRN evolution: https://amaarora.github.io/posts/2025-09-21-rope-context-extension.html — Aman Arora’s blog. Read this after the RoPE paper; save yourself weeks.

5. Chinchilla — Hoffmann et al., 2022 (“Training Compute‑Optimal Large Language Models”)

arXiv: https://arxiv.org/abs/<phone_number_or_numberic_id_or_random_id_21>

Extract: The scaling law that redirected the field. Given a compute budget C = 6·N·D (where N = params, D = tokens), the optimal split is N ∝ D — roughly 20 tokens per parameter. GPT‑3 was massively under‑trained by this measure; Chinchilla 70B trained on 1.4T tokens beat GPT‑3 175B.

Why you care as an inference engineer: this is why modern models are dense small (Llama‑3‑8B trained on 15T tokens, 1875 tokens/param — way past Chinchilla) and why MoE exists (decoupling total params from active params to bypass the Chinchilla constraint). Every model size decision downstream traces to this paper.

Read also: Kaplan et al. 2020 (original scaling laws, https://arxiv.org/abs/<phone_number_or_numberic_id_or_random_id_22>) — wrong on the exponents but historically foundational.

6. LLaMA 1 / 2 / 3 Reports — Meta, 2023 / 2023 / 2024

  • LLaMA 1 — https://arxiv.org/abs/<phone_number_or_numberic_id_or_random_id_23>: the model that opened the era. Establishes RMSNorm + RoPE + SwiGLU as the standard stack; trains 7B/13B/33B/65B on 1–1.4T tokens.

  • Llama 2 — https://arxiv.org/abs/<phone_number_or_numberic_id_or_random_id_24>: adds GQA at 34B/70B; ~2T tokens; the RLHF sections are the first serious open write‑up of RLHF at scale — read them.

  • Llama 3 — https://arxiv.org/abs/<phone_number_or_numberic_id_or_random_id_25>: 15T tokens (10× Llama 2!), 8B/70B/405B; the paper is the modern reference on training infrastructure.

What to extract from the Llama 3 report (this is the masterclass):

  • Section on training infrastructure: 16k H100 cluster, how they scheduled it, failure modes at scale (mean‑time‑between‑failures on that many GPUs is measured in hours).

  • 4D parallelism composition: TP × PP × CP × DP; exact configuration numbers.

  • Data curation pipeline (deduplication, quality filtering).

  • Post‑training pipeline: SFT → rejection sampling → DPO.

  • Long‑context extension section (from 8k to 128k via annealed continued pretraining + YaRN‑style RoPE scaling).

The Llama‑3 report is the single best free document on frontier training in existence. Read it three times over the course of this roadmap; extract more each time.

7. Fast Transformer Decoding: One Write‑Head Is All You Need — Shazeer, 2019 (MQA)

arXiv: https://arxiv.org/abs/<phone_number_or_numberic_id_or_random_id_26>

Extract: The origin of the KV‑cache‑reduction lineage. Noam Shazeer, in five pages, points out that during autoregressive decode, memory bandwidth of loading the KV cache dominates, and observes that you can share K and V across all query heads (n_kv = 1) with modest quality loss. Reduces KV cache by n_heads×.

Why still read: because the argument in Section 2 — arithmetic intensity of decode attention — is the cleanest derivation of “decode is memory‑bound” you will ever see. Six equations, one insight.

8. GQA: Training Generalized Multi‑Query Transformer Models from Multi‑Head Checkpoints — Ainslie et al., 2023

arXiv: https://arxiv.org/abs/<phone_number_or_numberic_id_or_random_id_27>

Extract: MQA (n_kv=1) loses too much quality on big models; MHA (n_kv=n_q) is wasteful. Grouped Query Attention: n_kv ∈ {n_q/8, n_q/4} — the sweet spot. Also shows you can uptrain an MHA checkpoint into GQA cheaply.

Numbers to internalise: Llama‑3–8B (n_q=32, n_kv=8 → 4×), Llama‑3‑70B (n_q=64, n_kv=8 → 8×), Llama‑3‑405B (n_q=128, n_kv=8 → 16×) — KV cache shrinks proportionally. This is what makes serving Llama‑3 at long context economically possible.

9. DeepSeek‑V2 (MLA) & DeepSeek‑V3 Reports — DeepSeek AI, 2024

DeepSeek‑V2: https://arxiv.org/abs/<phone_number_or_numberic_id_or_random_id_28> DeepSeek‑V3: https://arxiv.org/abs/<phone_number_or_numberic_id_or_random_id_29>

Extract: MLA (Multi‑head Latent Attention): instead of caching K and V per head, cache a compressed latent representation c_kv ∈ R^{d_c} (with d_c much smaller than n_kv · d_head) and up‑project on the fly. Also has a decoupled RoPE component to preserve position sensitivity through the compression. KV per token drops ~10× vs GQA, quality parity or better.

Why this matters more than GQA now: MLA is what makes serving 671B DeepSeek‑V3 at 128k feasible. Expect MLA (or an evolution) in every large frontier model going forward.

DeepSeek‑V3 report additional extracts (Phase 4/5/6 relevance):

  • FP8 mixed‑precision training at scale, with per‑block scaling and careful accumulate‑in‑fp32 discipline.

  • DualPipe: bidirectional pipeline parallelism reducing bubbles.

  • Auxiliary‑loss‑free MoE load balancing: bias trick instead of aux loss.

  • Multi‑Token Prediction (MTP): training‑time speculative‑decode friend.

  • Inference economics: explicit $ per million tokens numbers. Rare public candour.

Read the V3 report end‑to‑end at least once in Phase 1 (skim inference sections), once more in Phase 4 (understand disaggregation implications), and once more in Phase 6 (parallelism + FP8 training).

10. Bonus Companions (short, high‑value)

  • Neel Nanda / Anthropic “A Mathematical Framework for Transformer Circuits” — https://transformer-circuits.pub/2021/framework/index.html — not directly inference, but gives you the residual‑stream mental model that makes attention/MLP composition intuitive.

  • Kipply’s “Transformer Inference Arithmetic” — https://kipp.ly/transformer-inference-arithmetic/ — an entire post that is the arithmetic notebook you’re building, done by someone else. Compare notes.

  • Aleksa Gordić “Inside vLLM” — excellent deep dive; read at Phase 4, but bookmark now.

  • Llama 3 Herd Blog & GitHub — the model card and the llama-models repo docs supplement the paper.

The One‑Page Cheat Sheet

1. Attention Is All You Need                 → what attention IS
2. GPT-2/3                                   → decoder-only as default; scale as strategy
3. RoFormer (RoPE) + YaRN                    → position encoding of the modern era
4. Chinchilla                                → tokens/param is a first-class hyperparameter
5. Llama 1/2/3 reports                       → the modern stack, the training reality
6. MQA (Shazeer) + GQA                       → why KV cache shape is a design choice
7. DeepSeek-V2 (MLA) + V3                    → the current inference-frontier compression

Everything you read later — FlashAttention, PagedAttention, quantization methods, disaggregated serving — is a natural consequence of the tradeoffs these papers established. Read them cold, once, in Phase 1. You will re‑open them a dozen more times over the roadmap and each pass will feel new.