Phase 6 — Distributed Inference & Training at Scale

Months 13–18. This is where the middle of the ladder ends and the senior/staff work begins.

You have spent five phases learning to make one GPU sing. In this phase you learn to conduct an orchestra. The mental shift is smaller than it sounds — the roofline still rules, memory still bottlenecks, coalescing still matters — but a new axis appears: the network is a memory hierarchy too, and the choreography between compute, HBM, NVLink, and InfiniBand is the game.

Why this phase separates seniors from staff

Anyone who has read a vLLM tutorial can tell you what tensor parallelism is. Very few engineers can whiteboard, on demand:

• Why TP wants NVLink and PP tolerates InfiniBand (per-token communication volume).

• The two all-reduces per transformer block in Megatron TP, and where the column/row split cancellation happens.

• Why FSDP2 replaced FSDP1 (per-parameter sharding, cleaner mixed-precision, better CPU offload semantics).

• The 2(N−1)/N ring all-reduce bandwidth formula, and why tree all-reduce beats it at small message sizes.

• Why DistServe and Mooncake exist — the goodput argument, not the throughput argument.

• Why MoE breaks dense-model economics (capacity vs. active-compute split).

• Why KV-cache-aware routing at fleet scale beats any kernel win you can ship.

That list is your phase-6 exit exam. Everything in this folder is instrumentation for passing it.

Your unfair advantage (say this in studies)

You already think in terms of long-running distributed services: retries, backpressure, partial failure, tail latency, SLOs, multi-tenancy. Most people entering this field arrive from CUDA and have to learn that vocabulary painfully. You arrive from the other side — you have to learn what the distributed thing computes, not how to run a distributed thing. That is a much smaller gap.

The most valuable place that gap closes is rollout infrastructure — the training loop that contains an inference engine (vLLM inside verl / TRL / OpenRLHF). This is the current hot niche, and file 16_rollout_infra.md shows exactly why it maps onto your Zoho agentic-harness work with the labels changed.

The 5D parallelism taxonomy (memorize this before anything else)

Dim	What it splits	Communication	Locality wanted
TP (Tensor)	Individual matmuls	2 all-reduces / block (huge, per token)	NVLink domain
PP (Pipeline)	Layers into stages	1 send/recv per stage boundary (small)	Cross-node OK
DP (Data)	Batch across replicas	1 all-reduce of gradients per step	Any
FSDP (Sharded DP)	Params/grads/optim states	all-gather (fwd) + reduce-scatter (bwd) per layer	NVLink preferred
EP (Expert)	MoE experts across GPUs	2 all-to-alls per MoE layer	Fast fabric
CP/SP (Context/Sequence)	Sequence dimension	Ring pass of K/V	Fast fabric, long ctx only

You will compose these. Real training runs use DP × TP × PP (3D) or DP × TP × PP × EP (4D) or add CP for million-token training runs. Composition order matters because it dictates which comms happen on which fabric.

Files in this folder

File	Topic
01_parallelism_taxonomy.md	5D parallelism, communication cost per token, decision matrix
02_tensor_parallelism.md	Megatron column/row splits, derivation, when TP within-node
03_pipeline_parallelism.md	GPipe / 1F1B / interleaved / zero-bubble
04_fsdp_zero.md	ZeRO 1/2/3, FSDP2 API, PyTorch’s training workhorse
05_expert_parallelism.md	MoE serving, all-to-all cost, DeepEP, wide-EP
06_sequence_parallelism.md	Ring Attention for long context
07_nccl_networking.md	Ring vs tree all-reduce math, NVLink/IB/RoCE, topology
08_distributed_serving.md	TP/PP inference in vLLM/SGLang
09_disaggregated_serving.md	DistServe + Mooncake deep dive
10_kv_routing.md	Fleet-scale prefix-aware routing
11_ultra_scale_playbook.md	HF Ultra-Scale Playbook guided tour
12_scaling_book.md	Google DeepMind “How to Scale Your Model” tour
13_pretraining_small_model.md	124M–1B pretrain hands-on
14_modded_nanogpt.md	Speedrun community, Muon, PR history mining
15_fine_tuning_stack.md	LoRA / QLoRA / DPO / GRPO landscape
16_rollout_infra.md	verl / TRL + vLLM inside training — the hot niche
17_deepseek_v3_report.md	Full guided reading of DeepSeek-V3 tech report
18_projects.md	Phase 6 projects

Exit criteria (from base roadmap, sharpened)

1. Pretrained ≥124M model. Loss curve logged in W&B, every hyperparameter defensible on paper. You know why AdamW β₂=0.95 (not 0.999) for LLMs, why cosine schedule with warmup, why grad clip at 1.0, why bf16 not fp16.

2. Served a 70B-class model with TP=2 or TP=4. You have measured its scaling efficiency and can account for the gap between ideal and observed (all-reduce cost, kernel-launch overhead, imperfect load balance).

3. Whiteboard TP/PP/EP/FSDP comm patterns and costs. Given a model size, sequence length, batch size, and hardware topology, prescribe a parallelism strategy in five minutes and defend it.

4. You can explain when disaggregated PD pays and when it doesn’t. (Roughly: when prefill and decode SLOs diverge and traffic mix is skewed. Not when latency is dominated by a single phase.)

Big findings from this phase’s research (2024–2026)

• Mooncake officially won FAST ‘25 Best Paper — this validates KV-centric disaggregation as the reference architecture for large chatbot fleets. Kimi runs ~100M+ tokens/day on it. Repo: github.com/kvcache-ai/Mooncake.

• DeepSeek-V3 (Dec 2024) is the most instructive public LLM tech report ever written. MLA + auxiliary-loss-free MoE + MTP + fp8 training + expert parallelism, co-designed. If you read one paper this phase, read this twice.

• DeepSeek-R1 (Jan 2025) validated that pure RL from a base model can teach reasoning without SFT (R1-Zero). GRPO is now the default rollout algorithm; RL infrastructure has exploded as a field.

• Rollout infra is the current hot niche. FP8 rollout stacks in veRL, TIS/MIS importance-sampling corrections, vLLM-inside-training. Job postings with “verl” or “rollout” in the JD 6x’d in 2025.

• PyTorch FSDP2 is now the default for open-source training at 10B–70B scale. Megatron dominates 100B+. Deepspeed ZeRO-3 has quietly lost mindshare.

• Zero-bubble pipeline schedules (2023) + interleaved 1F1B (Megatron-2) are now the standard PP configurations for cross-node training.

• Wide-EP serving (DeepSeek-R1 class, 671B total / 37B active) requires expert parallelism ≥ 32 to be economical — the memory capacity math forces it.

Read files in numeric order. Do the exercises in 18_projects.md in parallel with reading — Phase 6 does not survive being consumed passively.