07 — NCCL & The Network Fabric

Why this file: Every parallelism strategy compiles down to a small vocabulary of collective operations. Their cost is not “network bandwidth” — it’s a specific function of algorithm × topology × message size. If you can’t derive the ring all-reduce cost formula, you can’t reason about TP scaling. If you don’t know when NCCL picks tree vs ring, you can’t debug the “why is my 8-GPU run slower than 4?” mystery.

Reading:

  • Sylvain Jeaugey (NVIDIA), Massively Scale Your Deep Learning Training with NCCL 2.4 (blog, 2019) — origin of the 2(N-1)/N formula popularization.

  • NCCL docs: https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/

  • Patarasuk & Yuan, Bandwidth Optimal All-reduce Algorithms for Clusters of Workstations (2009) — where the ring algorithm proof lives.

  • Baidu’s Bringing HPC Techniques to Deep Learning (2017 blog) — the paper that put ring all-reduce on the ML map.

The Collectives You Must Know Cold

Collective

Pattern

Uses

AllReduce

Sum/max across N ranks → identical result at each

DP grad sync, TP forward/backward

ReduceScatter

AllReduce then shard: rank i gets 1/N of the reduced result

ZeRO/FSDP grad reduction, Megatron-SP

AllGather

Each rank contributes 1/N, all ranks get the concatenated whole

FSDP param unshard, Megatron-SP

Broadcast

Rank 0 → everyone

Weight init, checkpoint scatter

Reduce

Sum to rank 0 only

Rare in training

AllToAll

Rank i’s j-th chunk → rank j’s i-th slot

MoE dispatch, sequence-parallel Ulysses

Send/Recv (point-to-point)

Directed message

PP microbatch handoff, Ring-Attention K/V pass

Key identity: AllReduce ≡ ReduceScatter + AllGather. This factorization is not academic — it’s how NCCL implements ring all-reduce, and it’s why FSDP (which uses the two halves separately) has the same total bytes moved as DDP AllReduce.

The Ring All-Reduce Cost Formula (Derive This From Memory)

Setup: N ranks, each holding a tensor of S bytes (say 1 GB of gradients each).

Ring algorithm:

  1. Logically split each rank’s tensor into N chunks of size S/N.

  2. Reduce-scatter phase: N-1 steps. In step k, rank i sends chunk (i-k mod N) to rank i+1 and receives chunk (i-k-1 mod N) from rank i-1, accumulating into its local copy. After N-1 steps, rank i owns the fully-reduced chunk (i+1 mod N).

  3. All-gather phase: N-1 more steps rotating the reduced chunks around the ring.

Per-link bytes: each step moves S/N bytes. Total steps 2(N-1).

$$ \boxed{\text{bytes per rank on the wire} = 2(N-1) \cdot \frac{S}{N} = 2S \cdot \frac{N-1}{N}} $$

As N → ∞, this approaches 2S per rank. That is the constant that dominates all your napkin math. Bandwidth utilization is bandwidth optimal — you cannot do better on a ring topology.

Time: t_allreduce ≈ 2S(N-1)/(N·B) + 2(N-1)·α where B is the slowest link’s unidirectional bandwidth and α is per-message latency.

  • The first term dominates for large S (bandwidth-bound).

  • The second term dominates for small S (latency-bound) — this is why NCCL fuses small collectives, and why you’ll see tree/double-tree at small message sizes.

Consequence for TP scaling

A TP=8 transformer block does 2 all-reduces of size B·L·H bytes per forward pass. With H100 NVLink4 ≈ 450 GB/s per GPU (bidirectional aggregate ~900 GB/s), an 8B hidden × batch × seq of ~256 MB per all-reduce costs ~0.5 ms — fine when GEMMs take 3-5 ms, catastrophic if you tried to run TP across PCIe (~64 GB/s ≈ 15x slower → all-reduce longer than compute).

This is why TP stays within an NVLink domain. Full stop.

Ring vs Tree vs Double-Tree

NCCL picks the algorithm based on message size and topology.

Algorithm

Bandwidth cost

Latency cost

Best for

Ring

Optimal 2(N-1)/N · S / B

2(N-1)·α — grows linearly with N

Large messages, in-node NVLink

Tree

~2S/B (higher, some links carry more)

2·log₂(N)·α — logarithmic

Small messages, latency-critical

Double-tree (NCCL default multi-node)

Near-ring bandwidth

Near-tree latency

Large-scale multi-node all-reduce

CollNet / SHARP

In-network reduction (IB switch does the sum)

O(1) hop

Frontier-scale, needs Mellanox HDR/NDR + SHARP

Rule of thumb: within a single NVLink domain, ring dominates. Across nodes with IB, double-tree or CollNet-SHARP wins.

The Fabric Reality (Order-of-Magnitude Table)

Fabric

Per-GPU bidi bandwidth

Typical latency

Where you meet it

NVLink 4 (H100)

~450 GB/s (900 GB/s NVSwitch aggregate)

sub-μs

Within 8-GPU node — TP, EP-dispatch

NVLink 5 (B200)

~900 GB/s per GPU (1.8 TB/s aggregate)

sub-μs

Blackwell nodes

PCIe Gen5 x16

~64 GB/s

~1-2 μs

Consumer boards, some cost-optimized H100 SXM variants without NVLink Bridge

InfiniBand NDR (400 Gb/s)

~50 GB/s per NIC

~1-2 μs

Cross-node in H100/B200 clusters (rail-optimized: 8 NICs per node, one per GPU)

RoCE v2 (Ethernet)

~25-50 GB/s per NIC (varies)

2-5 μs

Meta, MSFT clusters; hyperscaler alternative to IB

CX-7 400G RDMA over IB

50 GB/s

1 μs

Modern rail-optimized topology

The 10x cliff: intra-node NVLink is ~10x faster than inter-node IB. Every parallelism decision reduces to “which comm patterns can I keep on the fast side of the cliff?”

  • TP → NVLink only. Two all-reduces per block per direction × dozens of blocks per forward → totally unaffordable on IB.

  • PP → IB is fine. One send/recv of activations per stage boundary per microbatch. Tiny relative to compute.

  • DP/FSDP → IB is fine at scale. Grad sync happens once per step, can overlap with backward.

  • EP → NVLink preferred, but wide-EP crosses nodes. DeepSeek-R1 style wide-EP uses IB with heavy overlap (see DeepEP kernels).

Rail-Optimized Topology (What NVIDIA Reference Clusters Look Like)

A standard 8xH100 node:

GPU0 - GPU1 - GPU2 - GPU3
  \    |    /  \    |    /
   NVSwitch (450 GB/s per GPU, fully connected within node)
  /    |    \  /    |    \
GPU4 - GPU5 - GPU6 - GPU7
   |     |     |     |
  NIC0  NIC1  NIC2  NIC3    ← rail 0..3 (one InfiniBand NIC per GPU)
   |     |     |     |
   ├── Rail 0 IB switch ──┐
   └── Rail 1 IB switch ──┤   Each rail is its own leaf-spine network.
       Rail 2 IB switch ──┤   Cross-rail traffic must traverse a spine hop.
       Rail 3 IB switch ──┘

Key implication: NCCL is topology-aware. It uses NVLink for the intra-node portion of any collective, then IB per-rail for inter-node. Setting NCCL_TOPO_FILE or letting NCCL auto-detect via NCCL_TOPO_DUMP_FILE matters for large runs. Rails mean GPU_i on node A talks to GPU_i on node B on the same rail — never GPU_i→GPU_j across nodes without going up a spine.

The NCCL_DEBUG=INFO Ritual

When a distributed run hangs, silently corrupts, or scales like a brick, the first thing you do:

NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,COLL,ENV python train.py 2>&1 | tee nccl.log

What to look for in nccl.log:

  • NCCL INFO Bootstrap : Using <ifname>:<ip> — is it using IB or falling back to TCP over Ethernet?

  • NCCL INFO NET/IB : ... GID <n> — IB path is active. If you see NCCL INFO NET/Socket, you’re on TCP → 100x slower → fix it.

  • NCCL INFO Channel <k> — number of channels (parallel rings). Small message sizes use 2, large sizes 8-32.

  • NCCL INFO Ring <n> vs Tree <n> — which algorithm was selected.

  • NCCL INFO comm 0x... rank <r> nranks <N> ... commHash — verify all ranks joined the same communicator.

Common failure signatures:

  • Slower than expected + NET/Socket in log → NIC not detected, IB config wrong, fallback to TCP.

  • Hang at init with mismatched ranks → some processes died silently (check dmesg for OOM kills, torchrun logs).

  • Corruption on multi-node but not single → likely a PCIe→NIC affinity mismatch; force NCCL_IB_HCA=mlx5_0 etc.

  • Slow tail latency on IB → GID mismatch or wrong SL (service level); set NCCL_IB_GID_INDEX, NCCL_IB_SL.

Critical environment variables you WILL need:

  • NCCL_DEBUG=INFO|WARN — verbosity

  • NCCL_IB_DISABLE=1 — force TCP (debugging only)

  • NCCL_P2P_DISABLE=1 — force host-staged copies (rare, debugging)

  • NCCL_SOCKET_IFNAME=eth0 — pin the bootstrap interface

  • NCCL_IB_HCA=^mlx5_1 — exclude an HCA (rail isolation)

  • NCCL_ALGO=Ring|Tree — force algorithm (benchmarking)

  • NCCL_NVLS_ENABLE=1 — NVLink SHARP (Hopper+)

  • NCCL_ASYNC_ERROR_HANDLING=1 — make hangs fail fast rather than deadlock forever

Overlap: The Whole Game

At scale, compute overlaps communication or you lose. The specific overlaps:

  1. DDP: backward computes grads layer-by-layer; each layer’s grad kicks off an all-reduce while the next layer’s backward runs. PyTorch DDP does this automatically via bucket-fused all-reduce.

  2. FSDP: prefetch the next parameter all-gather during current forward compute; reduce-scatter current grad while next backward computes. Watch torch.profiler trace: you want NCCL streams solid, compute streams solid, minimal white space.

  3. TP+SP (Megatron-3): the reduce-scatter after MLP is followed by GEMM that can hide part of it. Recent Megatron does explicit torch.cuda.stream overlap.

  4. PP: the bubbles ARE the failure to overlap — 1F1B and Zero-Bubble are the schedule fixes.

  5. EP: all-to-all overlaps with expert compute in DeepEP.

Diagnostic: In Nsight Systems, colored NCCL rows should sit alongside compute rows, not sequentially. Gaps = perf left on the floor.

Napkin Math: How Much All-Reduce Time Are You Buying?

Say you’re training a 70B model with FSDP on 64 H100s across 8 nodes, IB NDR (50 GB/s per NIC).

  • Total params: 70B → 140 GB in bf16.

  • Grad reduce-scatter each step: 2·(N-1)/N · 140 GB ≈ 275 GB moved per rank aggregate across the phases (but split across N ranks, so per-rank on the wire is ~2·140/64 · 63 ≈ 4.3 GB).

  • At 50 GB/s per NIC (best case, hitting one rail): ~86 ms for the pure IB all-reduce phase.

  • FSDP splits it into per-layer overlapped chunks → target: hide most of it under backward compute.

If your measured step time is 500 ms and comm is 86 ms un-overlapped → 17% comm tax. Get it overlapped → <3% effective. This is why comm/compute overlap is where the money is.

AMD / ROCm Note (Awareness)

ROCm has RCCL — ABI-compatible with NCCL, same collective API. MI300X uses Infinity Fabric intra-node (~896 GB/s per GPU) and Broadcom/Marvell 400G NICs for inter-node. The math and mental model transfer 1:1; the constants change. If a customer says “we’re standardizing on MI300X for on-prem,” you’ll want ROCm literacy — the algorithms don’t change, only the fabric constants.

Exercises

  1. Derive the ring formula from scratch. No peeking. Write it on paper, plug in N=8, S=1GB, B=450 GB/s → predicted time. Then run nccl-tests/all_reduce_perf on a real 8xH100 node and see how close you got (usually within 30%).

  2. Force ring vs tree. With NCCL_ALGO=Ring then NCCL_ALGO=Tree, sweep message sizes from 1KB to 1GB. Plot bandwidth vs message size. Confirm the crossover point.

  3. Diagnose a TCP fallback. On any 2-node run, deliberately unset NCCL_SOCKET_IFNAME and let NCCL guess. Read the log. Find why it might pick a slow interface. Fix it.

  4. Kill a scaling curve. Train the same tiny GPT with DDP on 2, 4, 8 GPUs. Measure step time. Compute scaling efficiency. Explain the gap.