11 — The HuggingFace Ultra-Scale Playbook: A Guided Tour

What it is: A free, book-length treatment of 5D parallelism from the HuggingFace / nanotron team, distilled from thousands of real training runs on hundreds to thousands of GPUs. The best single free resource for distributed training and one of the best for distributed inference.

URL: https://huggingface.co/spaces/nanotron/ultrascale-playbook (verified). It’s hosted as an HF Space (Streamlit-style long-form doc) and updated over time. Print-friendly export available.

Companion repo: https://github.com/huggingface/nanotron (the actual training codebase the book documents).

---

1. Why this book exists and who it’s for

Every large-scale training story used to be locked behind FAANG-lab paywalls, delivered as tech reports (Llama, PaLM, GPT-3) with the interesting parts elided. Nanotron is HF’s open answer to Megatron, and the Ultra-Scale Playbook is its manual. It was written by people who actually ran the runs, not academics writing about them.

For you, its role in this roadmap is:

• Backbone reference for everything in this Phase 6 folder — cross-check my summaries against theirs.

• Concrete recipes — hyperparameters, mesh shapes, expected step times — so you don’t reinvent the wheel.

• Failure catalogue — the section on “things that break” is worth the whole book.

Read it after you’ve read files 01–10 of this folder. My summaries build the mental model; the playbook fills in operational reality.

---

2. Structure (approximate — the book is a living doc)

The playbook flows roughly like this. Section titles may drift as HF updates it; the mental map holds.

Part A — Single GPU, then scale up

1. The Single-GPU chapter. Kernels, memory hierarchy, activation checkpointing, mixed precision, gradient accumulation. Everything you already know from Phases 2–3. Read to calibrate their notation.

2. Data Parallel. DDP mechanics, all-reduce cost, gradient bucketing, computation/communication overlap. The “why does DDP scale sub-linearly?” answers live here.

Part B — Beyond one GPU’s memory

3. ZeRO / FSDP. Stages 1/2/3 with concrete memory formulas. FSDP2 API sketch. When to prefer each stage.

4. Tensor Parallelism. Megatron column/row splits. Sequence Parallelism inside TP (the Megatron-3 optimization). GQA-imposed TP caps.

5. Pipeline Parallelism. GPipe, 1F1B, Interleaved, Zero-Bubble. Bubble math. Real schedules from real runs.

Part C — Composition and Frontier

6. Context/Sequence Parallel. Ring Attention, Ulysses. How to fit 1M-token pretrain.

7. Expert Parallel. MoE routing, aux losses, all-to-all cost, load imbalance war stories.

8. 5D composition. How to pick a mesh: (DP, FSDP, PP, TP, CP). Decision trees.

9. Communication. NCCL internals from a user’s perspective, topology awareness.

Part D — Operational reality

10. Numerics and stability. Loss spikes, gradient clipping, mixed precision debugging, FP8 traps.

11. Checkpointing and resume. Async checkpointing, distributed checkpoint format, sharded state.

12. Profiling and debugging. Nsight, torch.profiler, NCCL logs, common hangs.

13. Cost estimation and the roofline for training. How to predict time-to-completion before spending $1M of GPU hours.

---

3. Chapters to read first (the 20% for 80%)

If you have a weekend, read these:

1. Data Parallel + FSDP chapters — the working reality of 90% of training runs today.

2. Tensor Parallel + Sequence Parallel — the section that made me finally understand the Megatron-3 all-gather/reduce-scatter trick.

3. 5D composition — the mental map of how to pick your parallelism shape.

4. Numerics / stability — worth its weight in gold; explains why your loss went NaN at step 8127.

5. Profiling — the operational chapter. Copies of screenshots you’ll re-derive on your own runs.

Skim the rest. Come back when you hit a specific problem.

---

4. Exercises the book invites (do them)

The playbook is interactive-ish; each section has questions embedded in prose. Formalize these into a lab notebook:

1. Predict-then-measure. For every configuration they show, predict the step time from first principles (params × 6 × tokens / peak FLOPs × MFU). Check against their measured numbers. Your predictions should be within 20%; if not, you missed a factor.

2. Rebuild their FSDP memory table. Given a 7B model, 4k context, batch 4/GPU, activation checkpointing on, compute weights/gradients/optimizer/activation memory for FSDP-1/2/3 by hand. Match their table cell for cell.

3. Design a mesh for a specific target. “Train Llama-3-70B on 256 H100s to Chinchilla optimality.” Pick DP/FSDP/TP/PP/CP shape; justify each choice against the book’s decision rules; sanity-check against the Llama-3 paper’s actual choices.

4. Reproduce one failure. Pick one “things that break” story and reproduce the pattern on a 2-GPU toy run (e.g., FSDP + gradient accumulation + BF16 without master weights → loss stagnation).

---

5. How to use it with this folder

• Read my 01_parallelism_taxonomy.md → then their Part B chapters. Reconcile the taxonomies (they use slightly different naming for CP vs SP).

• Read my 02_tensor_parallelism.md → then their TP chapter for the SP optimization I gestured at.

• Read my 04_fsdp_zero.md → then their FSDP chapter for FSDP2 API detail.

• Read my 07_nccl_networking.md → then their communication chapter for advanced topology tricks.

Where we disagree, prefer the book. Where a topic isn’t in the book, use mine (they de-emphasize serving; I don’t).

---

6. Adjacent HF resources worth knowing

• Nanotron repo — https://github.com/huggingface/nanotron. Read nanotron/parallel/tensor_parallel/ alongside file 02 of this folder.

• HF accelerate — the friendlier training launcher; supports FSDP2, DeepSpeed, Megatron backends. accelerate launch is your local training UX.

• HF transformers distributed docs — how the modeling code hooks into these mechanics.

• The HF picotron mini-tutorial repo — a compact educational implementation of nanotron; read it before the full nanotron codebase.

---

7. What the book doesn’t cover (yet)

• Inference serving in depth (vLLM/SGLang/PD-disagg) — that’s what my files 08–10 are for.

• Frontier RL rollout infra (verl-style) — file 16.

• Quantization for training beyond FP8 — thin here.

Complement the book with the DeepSeek-V3 report (file 17) and MLSys 2025 serving papers.

---

8. References

• Ultra-Scale Playbook (main): https://huggingface.co/spaces/nanotron/ultrascale-playbook

• Nanotron repo: https://github.com/huggingface/nanotron

• Picotron (mini educational nanotron): https://github.com/huggingface/picotron

• HF accelerate: https://github.com/huggingface/accelerate

Next: 12_scaling_book.md — the JAX/DeepMind counterpart.