03 — Papers Canon¶

Reading papers is not optional at PhD depth. Courses teach you the consensus view of ideas that were settled five years ago. Papers are where you encounter ideas that haven’t been settled yet — and where you learn to judge quality of evidence rather than just absorb conclusions. The 28 papers below are the ones you will encounter in every serious ML conversation. Not knowing them is a signal. Knowing them opens the door.

What most people get wrong: They read paper abstracts and conclusions, then claim they’ve “read the paper.” That’s reading the marketing material. The contribution lives in the method section, the ablations, and the limitations. If you haven’t read those, you haven’t read the paper.

How to Read a Paper (The Three-Pass Method)¶

First pass (15 min): Title, abstract, section headers, conclusions. Decide if it’s worth reading.
Second pass (1 hr): Full paper, skip proofs. Draw the main architecture. Understand the experimental setup.
Third pass (4+ hrs): Virtually re-implement mentally. Challenge every assumption. Run the ablations in your head.

For the canon papers below, aim for a second-pass read minimum. For papers in your primary research area, do the third pass.

Category 1: Foundations — The Papers That Define the Field¶

F1. Attention Is All You Need¶

arXiv: 1706.03762 | Year: 2017 | Authors: Vaswani et al. (Google Brain)

Why it’s in the canon: Introduced the Transformer architecture. Every major model in production today — BERT, GPT, T5, PaLM, LLaMA, Gemini — is a Transformer or Transformer derivative. If you don’t understand this paper, you don’t understand modern ML.

What to focus on: Multi-head attention mechanism (Eq. 1), positional encoding, the encoder-decoder structure, and why attention replaces recurrence.

Roadmap phase: Month 3-4

F2. Deep Residual Learning for Image Recognition (ResNet)¶

arXiv: 1512.03385 | Year: 2015 | Authors: He et al. (Microsoft Research)

Why it’s in the canon: Skip connections. Solved the vanishing gradient problem for very deep networks. ResNets enabled training 100+ layer networks and won ImageNet 2015. The residual connection concept appears in transformers, diffusion models, and nearly every modern architecture.

Roadmap phase: Month 3

F3. Batch Normalization: Accelerating Deep Network Training¶

arXiv: 1502.03167 | Year: 2015 | Authors: Ioffe, Szegedy (Google)

Why it’s in the canon: BatchNorm is in virtually every deep learning implementation. Understanding why it works (and why it sometimes doesn’t) is essential for debugging training instability.

Roadmap phase: Month 3

F4. Dropout: A Simple Way to Prevent Neural Networks from Overfitting¶

JMLR: Srivastava et al. 2014 | Year: 2014

Why it’s in the canon: The canonical regularization technique. Conceptually simple, theoretically interesting (ensemble interpretation), and still in use 10 years later.

Roadmap phase: Month 2

F5. Adam: A Method for Stochastic Optimization¶

arXiv: 1412.6980 | Year: 2014 | Authors: Kingma, Ba

Why it’s in the canon: The default optimizer for deep learning. Understanding the update rule (adaptive learning rates per parameter, first and second moment estimates) explains why Adam outperforms SGD on most tasks and when it doesn’t.

Roadmap phase: Month 2

Category 2: Architectures — The Structures That Power Modern Systems¶

A1. BERT: Pre-training of Deep Bidirectional Transformers¶

arXiv: 1810.04805 | Year: 2018 | Authors: Devlin et al. (Google)

Why it’s in the canon: BERT introduced masked language modeling and demonstrated that a single pretrained model could be fine-tuned to outperform task-specific models across 11 NLP benchmarks simultaneously. The “pretrain → fine-tune” paradigm that defines modern NLP.

Roadmap phase: Month 6

A2. Language Models are Few-Shot Learners (GPT-3)¶

arXiv: 2005.14165 | Year: 2020 | Authors: Brown et al. (OpenAI)

Why it’s in the canon: Demonstrated that scale alone enables emergent capabilities. In-context learning (no gradient updates) as a general-purpose task-solving mechanism. 175B parameters. Changed how the field thought about the relationship between scale and capability.

Roadmap phase: Month 7

A3. An Image is Worth 16x16 Words (ViT)¶

arXiv: 2010.11929 | Year: 2020 | Authors: Dosovitskiy et al. (Google Brain)

Why it’s in the canon: Applied the Transformer architecture directly to images (no convolutions). Showed that with enough data, pure attention outperforms CNNs. Launched a wave of vision transformer research.

Roadmap phase: Month 5

A4. CLIP: Learning Transferable Visual Models From Natural Language Supervision¶

arXiv: 2103.00020 | Year: 2021 | Authors: Radford et al. (OpenAI)

Why it’s in the canon: Contrastive pretraining on 400M image-text pairs. Zero-shot image classification. The architecture that connects vision and language, foundational to DALL-E, Stable Diffusion, and modern multimodal systems.

Roadmap phase: Month 8

A5. LLaMA: Open and Efficient Foundation Language Models¶

arXiv: 2302.13971 | Year: 2023 | Authors: Touvron et al. (Meta)

Why it’s in the canon: Open-weights LLM that democratized LLM research. Showed that smaller models trained on more tokens outperform larger models on fewer tokens (Chinchilla scaling). Spawned the open-source LLM ecosystem (Alpaca, Vicuna, Mistral, etc.).

Roadmap phase: Month 8

A6. Mistral 7B¶

arXiv: 2310.06825 | Year: 2023 | Authors: Jiang et al. (Mistral AI)

Why it’s in the canon: Grouped-query attention (GQA), sliding window attention (SWA). Outperforms LLaMA 2 13B on most benchmarks at 7B parameters. The paper showing the efficiency gains of architectural refinements over raw scale.

Roadmap phase: Month 8

Category 3: Training Methods — How to Make Models Actually Learn¶

T1. Training Language Models to Follow Instructions with Human Feedback (InstructGPT)¶

arXiv: 2203.02155 | Year: 2022 | Authors: Ouyang et al. (OpenAI)

Why it’s in the canon: Introduced RLHF (Reinforcement Learning from Human Feedback) as applied to LLM alignment. The technical foundation for ChatGPT. Understanding SFT → reward model → PPO pipeline is essential for LLM work.

Roadmap phase: Month 9

T2. LoRA: Low-Rank Adaptation of Large Language Models¶

arXiv: 2106.09685 | Year: 2021 | Authors: Hu et al. (Microsoft)

Why it’s in the canon: The dominant method for parameter-efficient fine-tuning of LLMs. Instead of updating all weights (billions of parameters), LoRA trains low-rank decompositions injected into attention layers. Makes fine-tuning LLMs feasible on consumer GPUs.

Roadmap phase: Month 9

T3. Scaling Laws for Neural Language Models¶

arXiv: 2001.08361 | Year: 2020 | Authors: Kaplan et al. (OpenAI)

Why it’s in the canon: The empirical foundation for understanding how model capability scales with compute, data, and parameters. The paper that justified GPT-3’s scale and influenced every subsequent LLM training decision.

Roadmap phase: Month 7

T4. Training Compute-Optimal Large Language Models (Chinchilla)¶

arXiv: 2203.15556 | Year: 2022 | Authors: Hoffmann et al. (DeepMind)

Why it’s in the canon: Corrected Kaplan et al. on optimal token-to-parameter ratios. Chinchilla showed that most LLMs at the time were undertrained — you should train a smaller model on more tokens, not a bigger model on fewer tokens. Directly influenced LLaMA’s design philosophy.

Roadmap phase: Month 7

T5. Deep Double Descent: Where Bigger Models and More Data Hurt¶

arXiv: 1912.02292 | Year: 2019 | Authors: Nakkiran et al. (OpenAI)

Why it’s in the canon: The double descent phenomenon — validation loss can increase then decrease again as model size grows past a threshold. Overturns the classical bias-variance tradeoff intuition. Fundamental for understanding over-parameterized models.

Roadmap phase: Month 5

Category 4: Generative Models — The Architecture Family Powering the Current Era¶

G1. Generative Adversarial Networks (GANs)¶

arXiv: 1406.2661 | Year: 2014 | Authors: Goodfellow et al.

Why it’s in the canon: The original GAN paper. Two-network adversarial training. Despite being largely supplanted by diffusion models for generation quality, the GAN framework is foundational and appears throughout the literature.

Roadmap phase: Month 5

G2. Auto-Encoding Variational Bayes (VAE)¶

arXiv: 1312.6114 | Year: 2013 | Authors: Kingma, Welling

Why it’s in the canon: The VAE is the reparameterization trick made concrete. Latent space representations, ELBO, encoder-decoder for generation. Conceptually foundational for understanding modern latent diffusion models.

Roadmap phase: Month 5

G3. Denoising Diffusion Probabilistic Models (DDPM)¶

arXiv: 2006.11239 | Year: 2020 | Authors: Ho et al. (Google Brain)

Why it’s in the canon: The paper that made diffusion models practical. Forward noising process, reverse denoising, the connection to score matching. Foundation of Stable Diffusion, DALL-E 2/3, Midjourney’s underlying architecture.

Roadmap phase: Month 8

G4. High-Resolution Image Synthesis with Latent Diffusion Models (Stable Diffusion)¶

arXiv: 2112.10752 | Year: 2021 | Authors: Rombach et al. (LMU Munich / Stability AI)

Why it’s in the canon: Latent diffusion — running the diffusion process in a compressed latent space rather than pixel space. Makes high-resolution generation computationally feasible. The paper behind every open-source image generation tool.

Roadmap phase: Month 8

Category 5: Efficiency & Production — Making Systems Run in the Real World¶

E1. Efficient Estimation of Word Representations in Vector Space (Word2Vec)¶

arXiv: 1301.3781 | Year: 2013 | Authors: Mikolov et al. (Google)

Why it’s in the canon: Word embeddings as the pretraining concept. Skip-gram and CBOW. The conceptual ancestor of all modern embedding techniques, BERT pretraining, and retrieval-augmented generation.

Roadmap phase: Month 4

E2. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness¶

arXiv: 2205.14135 | Year: 2022 | Authors: Dao et al. (Stanford)

Why it’s in the canon: Attention is O(n²) in sequence length. FlashAttention makes exact attention 2-4x faster through tiling and IO-aware computation. Now the default attention implementation in PyTorch and every serious LLM training stack.

Roadmap phase: Month 10

E3. QLoRA: Efficient Finetuning of Quantized LLMs¶

arXiv: 2305.14314 | Year: 2023 | Authors: Dettmers et al. (UW)

Why it’s in the canon: 4-bit quantization + LoRA. Enables fine-tuning a 65B LLM on a single 48GB GPU. The paper that made consumer/academic fine-tuning of large models viable. Most fine-tuning tutorials reference this.

Roadmap phase: Month 9

E4. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers¶

arXiv: 2210.17323 | Year: 2022 | Authors: Frantar et al. (IST Austria)

Why it’s in the canon: 3-4 bit weight quantization with minimal accuracy loss. The foundation for running quantized LLMs at inference. Every llama.cpp, GGUF, and quantized HuggingFace model uses a variant of this approach.

Roadmap phase: Month 10

Category 6: Alignment, Safety & Modern Paradigms¶

AL1. Constitutional AI: Harmlessness from AI Feedback¶

arXiv: 2212.08073 | Year: 2022 | Authors: Bai et al. (Anthropic)

Why it’s in the canon: Constitutional AI uses a set of principles (the “constitution”) to have the model self-critique and revise its outputs before RLHF. Introduced RLAIF (RL from AI Feedback). The technical foundation for Claude’s alignment approach.

Roadmap phase: Month 11

AL2. Self-Instruct: Aligning Language Models with Self-Generated Instructions¶

arXiv: 2212.10560 | Year: 2022 | Authors: Wang et al. (UW)

Why it’s in the canon: Using the model to generate its own instruction-following training data. The approach behind Alpaca (52K instruction examples from GPT-3). Democratized instruction-tuning for researchers without access to expensive human labeling.

Roadmap phase: Month 9

AL3. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (RAG)¶

arXiv: 2005.11401 | Year: 2020 | Authors: Lewis et al. (Facebook AI)

Why it’s in the canon: The original RAG paper. Augmenting LLM generation with retrieved documents to ground responses in external knowledge. The architecture behind every enterprise chatbot, search augmentation, and “chat with your documents” application.

Roadmap phase: Month 10

Roadmap Phase Summary¶

Phase	Months	Papers to Read
Foundations	1-3	F4, F5, F2, F3
Core DL	2-4	F1 (Attention), F2 (ResNet), E1 (Word2Vec)
Architectures	4-6	A1 (BERT), A3 (ViT), T5 (Double Descent)
Generative + LLMs	6-8	G1-G4, A2 (GPT-3), A5 (LLaMA), A4 (CLIP), A6 (Mistral)
Training Theory	7-8	T3, T4 (Scaling Laws + Chinchilla)
Alignment + Efficiency	9-11	T1 (InstructGPT), T2 (LoRA), E3 (QLoRA), E2 (FlashAttention), E4 (GPTQ), AL1-AL3

Paper Reading Tools¶

Semantic Scholar: semanticscholar.org — free citation graphs
Connected Papers: connectedpapers.com — visual paper relationship maps
Papers With Code: paperswithcode.com — every paper linked to its GitHub implementation
Elicit: elicit.com — AI research assistant for literature search

Return to README.md · Previous: 02_courses_canon.md · Next: 04_tools_and_libraries_canon.md