14 — modded-nanogpt: the Speedrun Community

1. What the speedrun is

A fixed, honest benchmark:

  • Task: train a language model on FineWeb until it reaches 3.28 cross-entropy validation loss.

  • Hardware: 8 × NVIDIA H100 (single node).

  • Baseline: Karpathy’s llm.c GPT-2 replication reaches this loss in ~45 minutes.

  • Current record (Jan 2025 snapshot): ~3 minutes (a >15× speedup). The record keeps falling.

  • Rule: train/val token splits are fixed. Everything else — architecture, optimizer, data ordering, precision — is fair game.

This is the modern lab notebook for how the frontier trains transformers. Every PR is a controlled experiment with a measured wall-clock delta, reviewed publicly. It is a better education than any course.

2. Why you should study it (even if you never submit a PR)

Speedrun optimizations are the same optimizations frontier labs use, just visible in the open. Reading the PR history is like reading Anthropic’s or Google’s private optimization notebook. Categories that recur:

Category

Example techniques

Optimizer

Muon; hybrid Adam+Muon; Nesterov momentum; QK-clip (Kimi K2)

Precision

bf16 everywhere; FP8 matmuls on H100 tensor cores; stochastic rounding

Architecture

RoPE tweaks; ReLU² / SwiGLU / SquaredReLU MLP; QK-norm; “softpick” attention gating; embed-sharing

Init & LR

µP-style init; Muon-specific init; per-parameter LR groups; layer-wise LR

Data

TokenMonster/BPE-alternative tokenizers; carefully-ordered data; deduplication

Compilation

torch.compile modes; CUDA graphs; custom Triton kernels for attention/MLP

Parallelism

Sequence packing; long-context via document masking; tensor parallel-ish overlap

Numerics

Loss-scaling for FP8; per-block scaling; skip-connection normalization

When you see a technique appear on the leaderboard, chase its PR discussion. That is where the actual ML happens.

3. Muon — the optimizer that started the modern speedrun era

Muon is Keller Jordan’s variant of SGD with momentum that adds an orthogonalization step to the update matrix before applying it.

The one-line version

Muon = SGD-with-momentum + “orthogonalize the momentum buffer using 5 Newton-Schulz iterations before the parameter update.”

# Pseudocode
def muon_step(param, grad, momentum, lr, beta=0.95):
    momentum = beta * momentum + grad
    ortho = newton_schulz(momentum, iters=5)   # ≈ U V^T where momentum = U S V^T
    param -= lr * ortho
    return momentum

The intuition

  • SGD/AdamW updates rows and columns unequally. In a matrix update ΔW = -lr · G, the SVD of G has some singular values much larger than others; those directions get “too much” update and dominate the trajectory.

  • Orthogonalizing replaces U S V^T with U V^T — all singular values become 1. Every direction in the momentum gets equal weight.

  • Newton-Schulz approximates that orthogonalization cheaply in bf16 without an actual SVD.

This is closely related to approximate second-order / natural-gradient methods (Shampoo, K-FAC family), but implemented as a per-matrix postprocess with negligible overhead.

The measured results

  • 1.35× wall-clock speedup on the NanoGPT speedrun vs a tuned AdamW.

  • 2.6 A100-seconds to 94% CIFAR-10 accuracy (down from 3.3).

  • The FLOP overhead of Muon at 124M scale is ~0.7% (5 Newton-Schulz steps × 768 hidden / 524k tokens per batch).

  • Now used or adapted in production: MuonClip (Kimi K2, adds QK-clip for stability at 1T-param scale).

Where Muon does not apply

  • 1D parameters (biases, LayerNorm gains, embeddings) — no matrix structure; use AdamW for these. The canonical Muon setup is hybrid: Muon on 2D matmul weights, AdamW on the rest.

  • Very small models (<1M params) — the constant overhead dominates.

  • Extremely large batch sizes — the argument for orthogonalization weakens when the momentum buffer is already well-conditioned.

Papers/posts to read (in order)

  1. Muon blog post (Jordan, 2024): kellerjordan.github.io/posts/muon/ — the origin document.

  2. Deriving Muon (Bernstein, 2025): jeremybernstein.github.io/deriving-muon — the mathematical justification via steepest descent under a spectral norm.

  3. Kimi K2 report (Moonshot, 2025, arxiv:<phone_number_or_numberic_id_or_random_id_153>): production use of MuonClip on a 1T MoE. This validates Muon at scale.

  4. Optimizer comparison for NanoGPT speedrunning (linked from Muon blog) — empirical comparisons vs Shampoo, Sophia, AdamW, Lion.

4. Mining the PR history as a curriculum

The repo README lists every record with the commit that produced it. Every one is worth reading. Suggested reading list:

Record milestone

Key innovation

What to steal

Baseline (llm.c, ~45 min)

Karpathy’s clean PyTorch trainer

The reference. Read every line.

Early Muon record

Muon optimizer introduced

The optimizer + hybrid AdamW-for-1D pattern

FP8 tensor cores

FP8 matmuls with per-block scaling

How to actually use H100 FP8 in a training run

Attention gating / softpick

Learnable per-head gates

Small architectural changes with outsize effect

µP-style init

Per-block/depth scaling of init

Correct init is a real speedup, not just “nicer”

Long context via document masking

Pack multiple docs into one 1024-seq

Sequence packing without cross-doc leakage

CUDA graphs / torch.compile modes

Kernel launch overhead removal

Especially important for small-batch decode

TokenMonster (alexjc, out-of-rules)

Alternative tokenizer

Reminder that tokenization is fair game and undervalued

Discipline for reading a PR:

  1. Read the PR description first — the wall-clock delta and the hypothesis.

  2. Read the diff before the discussion. Predict what the change does.

  3. Read the discussion — someone smart usually asks the right skeptical question.

  4. Note whether the technique generalizes (e.g., “does this help at 1B, or is it a 124M-specific overfit?”).

5. How to participate (even lightly)

Entering the leaderboard is a portfolio-tier achievement. But even the participation gradient below is career-valuable:

  1. Reproduce the current record on 8×H100. Rent for one day (~$50–100). Confirm you hit 3.28 in the claimed wall-clock. This alone teaches you more than a semester of ML class.

  2. Reproduce with one change ablated (e.g., turn Muon off; use AdamW). Report the loss curve and wall-clock difference. Post it as a blog write-up. This is a real experimental artifact.

  3. Fork and try one variation: a different attention scaling, a different LR schedule, a different init. Even if it doesn’t beat the record, the methodology of proposing → running → reporting is the skill.

  4. Submit a PR — only after 1–3 have made you fluent. Community norms are strict about honest benchmarks.

6. Things that transfer directly to your career

Even if you never train a frontier model, the speedrun community teaches:

  • Wall-clock is the metric that matters. Not FLOPs, not “steps to X loss.” Actual seconds. This is the same as “tokens/sec-goodput” in serving.

  • Every optimization must be measured, not argued. Every PR has a number. Every rollback has a reason. This is how Zoho engineering should treat serving optimizations too.

  • Small, controlled changes compound. No PR is heroic. 30 small PRs beat 1 giant one. Same as production engine work.

  • Public benchmarking builds trust. The speedrun community’s authority comes from reproducibility. Your career authority will come from the same discipline.

7. Adjacent communities and where the frontier is next

  • Sea AI Lab — Zero-Bubble PP (arxiv:<phone_number_or_numberic_id_or_random_id_154>): pipeline scheduling analog to the speedrun for large-scale training.

  • HuggingFace nanotron: production 5D parallelism codebase; speedrun-style optimizations at 1B–10B scale.

  • DeepMind Chinchilla / scaling-law community: the theoretical counterpart. Speedruns are laws-of-optimization; Chinchilla is laws-of-scale.

  • NVIDIA MLPerf training submissions: the corporate/closed version of the speedrun. Watching NVIDIA’s MLPerf training records is where GPU-specific optimizations become public.

  • The Nvidia Physics-informed / TransformerEngine team: where FP8 training tricks originate. TransformerEngine’s fp8 recipes eventually filter into speedrun PRs.

8. Exercises

  1. Reproduce the current record. Rent 8 × H100 for a day; run the tip of KellerJordan/modded-nanogpt; report wall-clock and val loss.

  2. Ablate Muon. Run the same code with AdamW substituted for Muon on the 2D params. Report the wall-clock delta and the loss-curve shape.

  3. Read three PRs deeply. For each, write a 200-word summary: hypothesis, change, measured delta, whether it generalizes to bigger models.

  4. Steal one trick for your Phase 6.13 pretrain. Apply one modded-nanogpt technique (e.g., Muon on 2D weights, or QK-norm) to your 124M baseline. Report if it helps.

  5. Derive Muon’s overhead for a hypothetical Llama-3-70B training run (m=8192, B=4M tokens/batch, T=5 NS iters). Comment on whether it stays negligible.

9. Where speedrun tricks go wrong at scale

A sobering note: not everything that helps a 124M in 3 minutes helps a 70B in 3 months.

  • Muon: validated at 1T scale (Kimi K2), likely production-ready.

  • FP8 training: validated (DeepSeek-V3), production-ready with careful scaling.

  • µP init: helps everywhere; production-ready.

  • Softpick / novel gates: mostly 124M-specific; treat as speculative for large runs.

  • TokenMonster / novel tokenizers: unclear; tokenizer choice at scale interacts with data curation in hard-to-predict ways.

  • Extreme LR schedules: helpful on toy scale, dangerous at production scale (loss spikes at 100B+ tokens have destroyed real runs).

Rule: speedrun tricks are hypotheses, not gospel. Validate at your scale before committing.

10. References

Next: 15_fine_tuning_stack.md — from pretraining to LoRA/QLoRA/DPO/GRPO.