04 — Stanford CS336: Language Modeling from Scratch (Your Backbone)

Verified Status (as of research date)

CS336 is the single best-fit university course for this entire roadmap. It is Percy Liang + Tatsunori Hashimoto’s Stanford course that has students build a GPT — tokenizer, transformer, FlashAttention in Triton, distributed training — from scratch. Assignments are open, code is on GitHub, lectures are on YouTube. No excuses.

What exists today

Edition	Status	URL
Spring 2025 (archived)	Full syllabus + lectures + assignments public	https://stanford-cs336.github.io/spring2025/
Spring 2026 (upcoming)	Announced online cohort Mar 30 – Jun 10, 2026	https://online.stanford.edu/courses/cs336-language-modeling-scratch
YouTube playlist	Stanford Online channel, full 2025 lecture set	https://www.youtube.com/watch?v=Rvppog1HZJY (Lecture 1)
Assignments repo	https://github.com/stanford-cs336 (5 assignment repos)	See below

Verdict: Use the Spring 2025 archive as your reference material and work assignments at your own pace. If timing aligns with your 13‑month track, the Spring 2026 live edition (April–June 2026) is a bonus but not required.

Community consensus (r/LocalLLaMA, HN, ML Twitter): “best course on LLMs online, period.”

The Instructors

• Percy Liang — CRFM director, foundation-models research lead, prior work on evals (HELM), retrieval (REALM), understanding scaling behavior.

• Tatsunori Hashimoto — Stanford NLP faculty, alignment and evaluation of LMs, LIMA / self-instruct lineage.

Both write papers you already read this chapter. Learning from them directly is a gift.

The Assignment Ladder (this is the whole game)

Five assignments, each a public GitHub repo with tests, scaffolding, and a writeup PDF. Do them in order — each builds on the last.

Assignment 1 — Basics

Repo: https://github.com/stanford-cs336/assignment1-basics

What you build:

• A byte-level BPE tokenizer from scratch (train on TinyStories or an OpenWebText subsample, save/load, encode/decode round-trip)

• A transformer from scratch in PyTorch — RMSNorm, RoPE, SwiGLU, GQA — matching modern Llama-class architecture (not the vanilla 2017 paper’s post-norm)

• Training loop with AdamW, cosine schedule, gradient clipping, mixed precision

• Small pretraining run on TinyStories (~30M model) to validate everything works

Environment: uses uv (Astral’s Rust-based Python package manager). Install it; it’s now the ML-community standard.

Time budget (honest): 40–80 hours if you’re serious. Skip nothing. The tests are strict.

This assignment alone gets you 80% of the Phase 1 exit criterion. If you had to do only one thing from CS336, do this.

Assignment 2 — Systems

Repo: https://github.com/stanford-cs336/assignment2-systems

What you build:

• FlashAttention-2 forward pass in Triton — this is the assignment. You will have earned this by the time Phase 3 arrives.

• Distributed training primitives — DDP, gradient bucketing, communication overlap

• Benchmarking harness — proper CUDA event timing, roofline placement

Where it fits in this roadmap: this is a Phase 3 assignment. Don’t rush it. Come back after finishing Phase 2 kernel work.

Assignment 3 — Scaling

Repo: https://github.com/stanford-cs336/assignment3-scaling

What you build:

• Compute-optimal training experiments — you literally run the scaling-law arithmetic

• Chinchilla-style token-budget derivations

Phase 6 material. Bookmark it.

Assignment 4 — Data

Repo: https://github.com/stanford-cs336/assignment4-data

What you build:

• Data pipeline: deduplication (MinHash/SimHash), quality filtering, PII scrubbing

• Small pretraining corpus construction

Phase 6 material.

Assignment 5 — Alignment

Repo: https://github.com/stanford-cs336/assignment5-alignment

What you build:

• SFT + DPO on a base model

• Understanding of the RLHF/GRPO loop as an infra pattern

Phase 6 material (bridges to the training-side of your inference expertise).

How to Use CS336 in This Roadmap

Don’t try to consume CS336 linearly in parallel with everything else. It’s dense, and half of it lives in Phases 3/6. Instead:

When	What to do
Phase 1, weeks 5–10	Assignment 1 (Basics), all of it. This is your Phase 1 capstone if you want a stronger backbone than the pure Karpathy path.
Phase 1, weeks 4–14	Watch Lectures 1–6 (overview, tokenization, architectures, MFU/roofline, GPUs, kernels-intro). Free intellectual multivitamin.
Phase 2–3	Assignment 2 (FA2 in Triton). Do this after Phase 2 kernel work, not before.
Phase 6	Assignments 3, 4, 5. By then your pretraining and distributed muscles will be built.
Continuous	Lectures as background listening while you code kernels. Percy and Tatsu’s asides are worth more than most other courses’ main content.

Lecture Anchors (the ones worth prioritising)

Watch these first — they map cleanly onto Phases 0–3 of this roadmap:

1. L01 — Overview + Tokenization — https://www.youtube.com/watch?v=Rvppog1HZJY — the mental model of “efficiency is everything in LM research”; kicks off the tokenization arc that ends in you shipping a BPE from scratch.

2. L02 — PyTorch, Resource Accounting — the arithmetic you did in 03_transformer_arithmetic.md, done again by Percy. Great re-derivation.

3. L03/04 — Architectures, MFU — pre-vs-post norm, RMSNorm/RoPE/SwiGLU/GQA justifications, model FLOPs utilisation as the real training metric.

4. L05 — GPUs — Stanford’s take on the GPU execution model. Watch after PMPP Ch. 1–4, not before.

5. L06 — Kernels & Triton — Triton primer, sets up Assignment 2.

Later lectures cover data, scaling laws, alignment, evaluation — all Phase 6 material.

What CS336 Doesn’t Cover (and where to fill)

CS336 is a training course with strong systems flavour. It does not cover:

• Production serving — vLLM/SGLang engine internals, continuous batching, paged KV → covered in Phase 4 of this roadmap.

• Quantization for deployment — GPTQ/AWQ/FP8 kernel work → Phase 5.

• llama.cpp / GGUF / local ecosystem → Phase 4.

• Enterprise deployment realities → Phase 7.

CS336 gets you fluent in the model + training half of the stack. This roadmap wraps the inference/serving/production half around it. That is the whole design.

Pragmatic Notes

• The uv requirement: install with curl -LsSf https://astral.sh/uv/install.sh | sh. Every assignment README assumes it.

• Compute needs for A1: a single 24GB GPU is enough for TinyStories training. The larger-scale exercises want more, but you can shrink them.

• Don’t google solutions: the tests will let you cheat past them. That will not fool an study partner who asks you to reproduce a working RoPE from memory on a whiteboard.

• The writeup PDFs: each assignment ships a writeup with derivations and expected numbers. Read them like papers. The FA2 writeup in A2 is particularly good.

Exit Test

By the time you leave CS336 (over the full 13-month arc, not week 1):

• Assignment 1 passing: required for Phase 1 exit.

• Assignment 2 (FA2 Triton) passing: required for Phase 3 exit.

• Assignments 3–5: strongly recommended for Phase 6 exit, especially A5 which teaches you the inference-inside-training pattern (rollout generation with vLLM) that is one of the hottest infra niches in 2025–2026.

CS336 is not one milestone. It is the backbone that carries you across five of them.