06 — Paper Reading Schedule (Month-by-Month, 13-Month Timeline)

Months in this file are aligned with the phase table in the base roadmap. Each month has primary reads (mandatory before the phase deliverable is doable) and companion reads (parallel-track, deepen the primary).

Budget: 3–4 papers/month, ~4–6 hours per paper (mix of easy and hard). Total ~50 hours/month on papers. Roughly 8–10 hrs/week of your 10–15-hour weekly budget.

Golden rule: do not read a paper before you’ve written the naive version of what it fixes. Papers explain fixes; the fix is only felt if you’ve felt the problem. This is why the schedule is intertwined with the projects, not front-loaded.

Month 1 (Phase 0/1 kickoff)

Primary reads:

  1. Attention Is All You Need (foundations #1) — [EASY]

  2. GPT-2 tech report (foundations #2) — [EASY]

  3. Kipply “Transformer Inference Arithmetic” (not a paper; https://kipply.substack.com/p/transformer-inference-arithmetic)

  4. Horace He “Making Deep Learning Go Brrrr” (not a paper; https://horace.io/brrr_intro.html)

Companion: Roofline paper (kernels #1) — read once now, again in Month 3.

Deliverable this month is coupled to: your from-scratch nanoGPT training run and CS336 A1 basics.

Month 2 (Phase 1 deep work)

Primary reads:

  1. RoPE / RoFormer (foundations #5) — [MEDIUM]

  2. Llama-1 paper (foundations #6) — [EASY]

  3. GPT-3 (foundations #3, skim) — [EASY]

  4. Chinchilla (foundations #4) — [MEDIUM]

Companion: Skim Llama-2 (foundations #7) for GQA arrival.

Coupled to: your inference-only from-scratch GPT (loads real weights, has KV cache).

Month 3 (Phase 1 close + Phase 2 open)

Primary reads:

  1. MQA — Shazeer 2019 (foundations #9) — [EASY, 6 pages]

  2. GQA (foundations #10) — [EASY]

  3. DeepSeek-V2 for MLA (foundations #11) — [HARD]

  4. Volkov “Better Performance at Lower Occupancy” (kernels #9) — [EASY]

Companion: PMPP chapters 1–5 (memory hierarchy, thread execution, tiled matmul).

Coupled to: SGEMM ladder started, first CUDA kernel compiles.

Month 4 (Phase 2 core)

Primary reads:

  1. Online softmax (kernels #2) — [EASY]

  2. Triton paper (kernels #8) — [MEDIUM]

  3. Simon Boehm SGEMM worklog (blog: https://siboehm.com/articles/22/CUDA-MMM) — read end-to-end while doing the ladder.

Companion: Reread Roofline. Skim ThunderKittens blog.

Coupled to: SGEMM ladder complete to ~80% cuBLAS; fused softmax and RMSNorm kernels done.

Month 5 (Phase 3 opens)

Primary reads:

  1. FlashAttention-1 (kernels #3) — [HARD]

  2. FlashAttention-2 (kernels #4) — [HARD]

Companion: Tri Dao’s GPU MODE lecture on FA (Lec 12 by Thomas Viehmann covers FA in depth); reread MQA/GQA.

Coupled to: Your first Triton FA2 implementation begins.

Month 6 (Phase 3 close)

Primary reads:

  1. FlashAttention-3 (kernels #5) — [HARD]

  2. Flash-Decoding blog (kernels #6) — [MEDIUM]

  3. ThunderKittens blog + paper (awareness) — [MEDIUM]

Companion: Skim CUTLASS/CuTe quickstart (kernels #10) — you don’t have to master it yet.

Coupled to: Triton FA2 numerics-correct + benchmarked vs flash-attn official.

Month 7 (Phase 4 opens)

Primary reads:

  1. Orca (engines #1) — [HARD]

  2. vLLM / PagedAttention (engines #2) — [HARD]

  3. Aleksa Gordić “Inside vLLM” (blog: https://www.aleksagordic.com/blog/vllm)

Companion: Read vLLM V1 architecture docs. Start reading vLLM source (scheduler + block manager).

Coupled to: Mini-engine design doc written; block manager stub coded.

Month 8 (Phase 4 middle)

Primary reads:

  1. Sarathi / Sarathi-Serve (engines #3) — [HARD]

  2. SGLang / RadixAttention (engines #4) — [HARD]

  3. FlashInfer (kernels #7) — [HARD]

Companion: Read SGLang radix_cache.py source. Read FlashInfer’s scheduler.cuh.

Coupled to: Mini-engine has continuous batching + paged KV working; ready to add prefix caching.

Month 9 (Phase 4 speculative decoding + capstone push)

Primary reads:

  1. Leviathan speculative decoding (engines #5) — [MEDIUM]

  2. Medusa (engines #6) — [MEDIUM]

  3. EAGLE-1/-2/-3 (engines #7) — [HARD] — read them in order, one per week.

Companion: Outlines paper (engines #9). Skim XGrammar (engines #10) if you’re integrating structured output.

Coupled to: Mini-engine capstone shipping. First blog post on benchmarks vs vLLM.

Month 10 (Phase 4 close + local stack + Phase 5 opens)

Primary reads:

  1. GGUF k-quants / llama.cpp docs (quantization #10) — [MEDIUM]

  2. LLM.int8() (quantization #1) — [MEDIUM]

  3. Massive Activations paper (arxiv:<phone_number_or_numberic_id_or_random_id_63>) — [MEDIUM]

Companion: XGrammar (engines #10) if not done last month. Reread PagedAttention now that you’ve implemented it.

Coupled to: Local llama.cpp + GGUF fluency; quantization bake-off begins.

Month 11 (Phase 5 core)

Primary reads:

  1. GPTQ (quantization #2) — [HARD]

  2. AWQ (quantization #3) — [MEDIUM]

  3. SmoothQuant (quantization #4) — [MEDIUM]

Companion: Marlin repo (quantization #9) — read README + main kernel file.

Coupled to: Quantization bake-off post shipping.

Month 12 (Phase 5 close + Phase 6 opens)

Primary reads:

  1. FP8 formats (quantization #6) — [EASY]

  2. OCP MX spec (quantization #7) — [MEDIUM]

  3. QLoRA (quantization #5) — [EASY]

  4. KIVI (quantization #8) — [MEDIUM]

  5. Megatron-LM (distributed #1) — [HARD]

Companion: HuggingFace Ultra-Scale Playbook — read Parts 1–3.

Coupled to: Naive W4A16 Triton kernel done; first FSDP2 training run begins.

Month 13 (Phase 6 core)

Primary reads:

  1. ZeRO (distributed #2) — [HARD]

  2. GPipe + PipeDream (distributed #3, #4) — [MEDIUM]

  3. GShard + Switch Transformer (distributed #5, #6) — [MEDIUM]

Companion: “How to Scale Your Model” (Google DeepMind) — Chapters 1–5. Reread Llama-3 infra sections.

Coupled to: Small pretrain (124M) running distributed; first TP=2 serving experiment.

Month 14 (Phase 6 serving + start Phase 7)

Primary reads:

  1. Ring Attention (distributed #7) — [HARD]

  2. DistServe (distributed #8) — [HARD]

  3. Mooncake (distributed #9) — [HARD]

Companion: LMCache paper + NIXL docs (distributed #10).

Coupled to: 70B TP serving deployment; PD-disaggregation experiment.

Bonus / capstone month (Month 13 if you’re on the compressed timeline, Month 15 if you’re at the 18-month pace)

Primary reads:

  1. DeepSeek-V3 tech report (distributed #11) — [HARD]

  2. DeepSeek-R1 (distributed #12) — [MEDIUM]

  3. Llama-3 report end-to-end reread (foundations #8 / distributed #13).

Companion: DeepEP paper (awareness). Verl repo walkthrough.

Coupled to: Reference-architecture doc written; the OSS-PR campaign begins in earnest.

The reading cadence rule (memorize it)

  • Weekdays: 30 minutes of code reading (see 13_discipline/04_reading_code_daily.md). Not papers.

  • Weekends: 4–6 hours of paper reading, split across 2 sessions. Never binge.

  • After each paper: 30 minutes of notes using the template in 09_papers/README.md.

  • Before each paper: skim the abstract of the paper that this paper cites first. Papers rarely exist alone.

Every 90 days: re-read your own notes on 3 papers you’ve forgotten. This is the actual review — nothing else works.

The “I’m behind schedule” recovery rules

You will fall behind. Everyone does.

  • 1 month behind: catch up on paper reading during a light project week. Do not skip.

  • 2 months behind: skip the [EASY] papers, keep [MEDIUM] and [HARD]. You lose vocabulary but keep depth.

  • 3+ months behind: audit and re-plan. If you’ve fallen 3 months behind while still shipping projects, your project ambition was too high, not your paper load. Cut a project.

Never skip the paper that motivates the current project. Reading Sarathi after your engine has ITL spikes is worthless — you’ll rediscover the wheel and lose the pedagogical value.

Papers that DIDN’T make the canon (and why)

The seed doc lists ~40 papers. I added a few (Volkov, XGrammar, LMCache, Massive Activations, EAGLE-3, DeepSeek-R1) and dropped or awareness-only-ed some. What’s not in the canon that you might expect:

  • BERT / T5 — encoder-only / encoder-decoder; not relevant to modern LLM serving.

  • Longformer / BigBird — sparse attention; obsoleted by long-context dense (RoPE-YaRN, ring attention).

  • ALiBi — lost to RoPE, awareness only.

  • RWKV / Mamba — fascinating alternative architectures, awareness only. Do not become an “SSM person” during the roadmap; that’s a niche after you’re hireable, not before.

  • Phi / SLM family — great models, no unique inference-engineering ideas.

  • StreamingLLM / attention sinks — clever but niche.

Add them after the 13 months.