01 — The Karpathy Path (Zero to Hero → nanoGPT → llm.c → nanochat)

“Type every line. Copy-pasting Karpathy is worse than not watching him.”

Andrej Karpathy’s teaching materials are the single best on-ramp to language modeling that exists in any medium. They are also easy to fool yourself with: watching passively feels productive but doesn’t build skill. The path below is what you actually do.

Verification status (as of research date):

Round 1: Zero to Hero (video lectures)

The playlist: https://karpathy.ai/zero-to-hero.html (also on YouTube: search “Karpathy Zero to Hero”).

Five videos, ~15 hours total. Type every line into your own Jupyter notebook. Don’t fork the repo, don’t clone the notebook. Type.

#

Video

What you build

Why it matters

1

The spelled-out intro to neural networks and backpropagation

micrograd (scalar autograd)

The chain rule as bookkeeping. Every PyTorch backward is this in a trench coat.

2

The spelled-out intro to language modeling (makemore part 1)

Bigram model

Softmax, cross-entropy, sampling. Bare-bones LM.

3

Building makemore Part 2: MLP

Character-MLP

Embeddings, batch dimension, train/dev/test split, hyperparam tuning.

4

Building makemore Part 3–4: Activations, gradients, batchnorm; WaveNet

Deeper MLPs, normalization

Why activations blow up; why batchnorm/layernorm exists. The soul of numerical stability.

5

Let’s build GPT: from scratch, in code, spelled out

nanoGPT-style tiny transformer

Attention as compute, positional embeddings, blocks, residuals. This is the video that changes how you think.

Bonus: Two follow-ups that are worth every minute:

  • “Let’s build the GPT Tokenizer” (~2 hours): BPE from scratch. Watch this before Round 2 build.

  • “Let’s reproduce GPT-2 (124M)” (~4 hours): a training-scale walkthrough, in nanoGPT style, culminating in reproducing GPT-2 124M on OpenWebText. This is the bridge to llm.c.

Protocol: for each video, work through in three passes:

  1. First pass (live): watch normally, type along.

  2. Second pass (24 hours later): rebuild the notebook from a blank file without watching. Watch only when stuck. This is where learning consolidates.

  3. Third pass (a week later): teach it to a hypothetical junior. If you can explain “why is softmax a normalization step and not just any function that sums to 1?”, you’ve earned the video.

Budget: 3–4 weeks at 5–6 hrs/week for the full series with the three-pass discipline.

Round 2: nanoGPT (the reference implementation)

Repo: https://github.com/karpathy/nanoGPT

What nanoGPT is: ~300 lines of PyTorch that trains and samples GPT-2-class models. Deliberately readable. The reference against which every custom implementation is checked.

Do this:

  1. Clone. Read model.py end to end. Don’t skim.

  2. Run python train.py config/train_shakespeare_char.py on your 3090. Aim for validation loss ~1.4 in ~10 minutes. This is the sanity check.

  3. Sample from the trained model with python sample.py. See it produce Shakespeare-shaped garbage. Feel the magic.

  4. Modify model.py to add rotary embeddings instead of learned positional embeddings. Retrain. Compare validation loss curves.

  5. Modify model.py to add SwiGLU instead of GELU. Retrain. Compare.

  6. This is where you graduate from nanoGPT: the codebase doesn’t do inference well (no KV cache), so move to Round 3.

Time budget: 3–5 days of evenings.

Warning: don’t waste weeks trying to “understand every line of the training loop.” Once you can explain the loss curve, the AdamW step, and the mixed-precision path, move on. The next thing (llm.c) will teach you more per hour.

Round 3: llm.c (the C/CUDA reproduction)

Repo: https://github.com/karpathy/llm.c

What llm.c is: the same GPT-2 model, in raw C (~1000 LOC in train_gpt2.c) and raw CUDA (train_gpt2.cu), no PyTorch. ~26k stars, actively developed as of late 2025. Karpathy’s famous benchmark: reproduce GPT-2 124M in 90 minutes for $20 on 8×A100 (discussion #481).

Verified status: actively maintained; the repo has weekly commits; the discussion tab is a treasure trove of engineering notes.

Why you read it now: llm.c is the shortest legible bridge from Python-level intuition to what actually happens on the GPU. It is 90% of what Phase 2’s SGEMM ladder will teach you, presented in a familiar context.

Do this:

  1. Read train_gpt2.c end to end. It’s ~1000 lines of C. Type answers to these questions into a notes file:

    • How is memory allocated for parameters, gradients, activations, KV cache? (Hint: one big arena.)

    • How is the AdamW optimizer implemented? (~10 lines.)

    • Where is the attention computed, and what’s its memory-traffic pattern?

  2. Read train_gpt2.cu. Note which kernels are called and where. Do not try to understand every kernel yet — you’ll come back in Phase 2.

  3. Run the training script if you have hardware (a single 3090 works fine for the 124M forward-only test). Compare tokens/sec to what Karpathy reports on 8×A100.

Time budget: 6–10 hours to read, plus optional overnight training runs.

Key discussion to read: #481 (“reproducing GPT-2 124M on 8×A100 in 90 minutes”) — an engineering journal of the last mile of training performance. This is the flavor of Phase 6.

Round 4: nanochat (the new capstone)

Repo: https://github.com/karpathy/nanochat

Released October 13, 2025, this is the current top of Karpathy’s teaching pipeline: a single ~8000-LOC repo that does the entire ChatGPT pipeline — tokenizer training, pretraining, mid-training, SFT, RL (GRPO), evaluation, inference server, and a web UI. The tokenizer is written in Rust (rustbpe) for speed.

Verified capability numbers (from repo README):

  • ~$100 tier: 4 hours on 8×H100, produces a depth-30 model with ~GPT-3 Small (125M) capability. Reported evals: 40s MMLU, 70s ARC-Easy, 20s GSM8K.

  • ~$300 tier: larger config, better numbers.

  • Has a speedrun leaderboard measured by DCLM CORE score — modern-nanoGPT-style community optimization.

  • Announced as the capstone project for Karpathy’s upcoming LLM101n course at Eureka Labs.

Why this matters for your roadmap: nanoGPT teaches pretraining. nanochat teaches the rest — SFT, RL, evals, inference server, web UI. All the things you’ll need to know for Phases 4–7. Reading nanochat now (Phase 1) gives you a mental map for what’s ahead.

Do this:

  1. Read the README. Read the codebase top-level. You don’t need to run it yet (though you can rent 8×H100 on RunPod for ~$25 and do a $100 speedrun — excellent portfolio piece).

  2. Understand the pipeline order: tokenizer → pretrain → mid-train → SFT → RL → eval → serve. This is your Phases 1, 4, 6, 7 sequenced in one repo.

  3. Save the RL / inference server sections for a re-read after Phase 4.

Time budget: 4–6 hours reading now; ~1 day + $100 if you do a full run later.

What NOT to Do

  • Do not get lost in Karpathy’s older Stanford CS231n materials — they’re excellent but pre-transformer.

  • Do not copy-paste his notebooks. Type. Every. Line.

  • Do not try to write your own “improved” version of nanoGPT before you’ve finished the Round 2 inference rebuild. It’s a productivity trap.

  • Do not confuse llm.c with a production trainer. It is a reference, glorious in its clarity. Real production training uses PyTorch + Megatron / nanotron / TorchTitan. You’ll meet those in Phase 6.

The Currency Check: Is nanoGPT Still Relevant?

Short answer: yes, but the ecosystem has expanded. As of late 2025:

Repo

Purpose

Status

Verdict

nanoGPT

Minimal GPT pretraining

~40k stars, occasionally updated

Still the reference; use for Round 2.

llm.c

Raw C/CUDA GPT-2 reproduction

~26k stars, active

Use as Round 3. Best bridge to Phase 2.

nanochat

Full pipeline: pretrain → SFT → RL → serve → UI

~30k stars, active

NEW canonical capstone. Use as Round 4.

modded-nanoGPT (Keller Jordan et al.)

nanoGPT speedrun with Muon optimizer, aggressive training tricks

~2k stars, active

Read its PR history in Phase 6 for a modern training-optimizations education.

picoGPT (Jay Mody)

GPT-2 inference in 60 lines of numpy

~3k stars, stable

Fun weekend read; not a substitute.

Use this pyramid: nanoGPT for pretraining intuition, llm.c for systems intuition, nanochat for full-pipeline intuition. Modded-nanoGPT is a Phase 6 side dish.

Exit Deliverable for This File

  1. karpathy_notebooks/ — your five Zero-to-Hero notebooks, typed by hand.

  2. nanogpt_experiments.md — a report on your RoPE and SwiGLU modifications with loss curves.

  3. llmc_notes.md — your annotated notes on train_gpt2.c, with answers to the three questions above.

  4. nanochat_map.md — a mental map of nanochat’s pipeline, showing which stage maps to which future phase.

When someone asks “can you build a GPT from scratch?” you should be able to say “I’ve built three, and here they are.”