08 — Distributed Inference in vLLM & SGLang

Why this file: Serving a model larger than one GPU (or wanting lower latency than one GPU allows) means TP/PP inside the inference engine. But the tradeoffs at serving-time are the opposite of training: batch is small, decode is memory-bound, and every all-reduce eats latency you can’t hide. Understanding when TP pays and when it destroys goodput separates people who can run vLLM from people who can architect inference clusters.

Reading:

The Serving TP Question

Training TP: activation-heavy, always compute-bound at prefill-size batches, all-reduce cost dominated by GEMM cost → TP=8 within-node scales at ~85-95% efficiency.

Serving TP at decode: each rank does 1/N of the GEMM (linear in FLOPs saved) and still pays 2 all-reduces per block (independent of TP degree in bytes moved). At batch=1 decode, the GEMM is already memory-bound and tiny — you’re spending time on 128×H all-reduces, not on 128×H·H matmuls. Scaling efficiency at low batch: 40-70% typical.

Rule of thumb (memorize this):

Regime

TP scaling efficiency

Why

Prefill (compute-bound)

85-95% within-node

GEMMs dominate, all-reduces overlap-ish

Decode, high batch (32+)

70-85%

GEMMs still meaningful, some overlap

Decode, batch=1

40-65%

All-reduce latency dominates; NVLink helps a lot

Cross-node TP

Don’t.

IB is 10x slower than NVLink → all-reduce longer than block itself

The corollary: TP is a capacity decision more than a latency decision. You use TP=2/4/8 to fit a model that doesn’t fit in one GPU. You don’t use TP for latency until you’ve saturated everything else — quantization, better kernels, larger batch.

When TP Actually Pays for Latency

Batch-1 decode of a 70B model in bf16 on ONE H100 (80 GB):

  • Weights alone: 140 GB → doesn’t fit. FP8: 70 GB → fits with room. INT4: 35 GB → plenty of room.

  • If fp16 required: TP=2 (2× H100, 160 GB total) is mandatory just to hold weights.

  • Decode speed on TP=2: theoretical 2 × HBM_BW / (params_per_gpu × bytes) = 2 × 3.35 TB/s / (35 GB × 2 bytes) ≈ 96 tok/s per user (fp16). Real: ~70-85 tok/s after all-reduce overhead.

  • Decode on TP=4: theoretical 190 tok/s, real ~130-150. Efficiency dropped from 100%→~75%.

  • Decode on TP=8: theoretical 380 tok/s, real ~200-260. Efficiency ~60-70%.

The scaling starts to break down because the all-reduce time (fixed per block) becomes a bigger fraction of the shrinking per-rank compute time. The right answer for 70B latency is usually TP=2 or TP=4 within a node, not TP=8.

Configuring TP in vLLM

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 4 \
  --pipeline-parallel-size 1 \
  --max-model-len 2412.19437 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --kv-cache-dtype fp8

Key knobs:

  • --tensor-parallel-size (TP): must divide num-attention-heads and num-kv-heads (for GQA models). Llama-3-70B has 8 KV heads → max TP=8 without KV-head replication.

  • --pipeline-parallel-size (PP): use only if you need cross-node. Adds latency.

  • --enable-expert-parallel (EP): for MoE models. TP × EP × PP × DP must equal world size.

  • --distributed-executor-backend {ray,mp}: ray for multi-node, mp (multiprocessing) for single-node.

Multi-node vLLM:

# On head node:
ray start --head
# On worker nodes:
ray start --address='<head_ip>:6379'
# Then serve with --tensor-parallel-size 8 --pipeline-parallel-size 2 for 16 GPUs across 2 nodes

Configuring TP in SGLang

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.3-70B-Instruct \
  --tp 4 \
  --dp 1 \
  --context-length 2412.19437 \
  --mem-fraction-static 0.85 \
  --enable-torch-compile \
  --kv-cache-dtype fp8_e5m2

SGLang-specific:

  • --dp runs multiple full replicas (data-parallel serving) — use for throughput scaling when a single replica is fast enough.

  • --tp × --dp = world size.

  • --attention-backend {flashinfer,triton,torch_native} — flashinfer is default, fastest for paged KV.

  • RadixAttention prefix cache is on by default — this is often the biggest single win over vLLM’s older implementations, especially for your agentic workloads.

PP for Serving: Almost Always the Wrong Answer

PP saves you from TP’s all-reduce cost by turning it into a single send/recv per stage. Sounds great. It’s a throughput win at high concurrency (pipelining microbatches through stages) but a latency disaster at low concurrency (each token waits for p stage passes).

Serving PP math: with PP=p stages, decode latency per token ≈ p × per-stage compute + p × comm. With TP=p, latency per token ≈ per-block compute + per-block all-reduce. TP wins on latency for interactive; PP only wins when:

  1. You literally can’t fit even TP=8 in one node (e.g., serving DeepSeek-R1’s 671B params).

  2. Cross-node is unavoidable → PP over IB, TP inside NVLink domain.

Standard wide-model recipe: TP=8 within each node, PP=N across N nodes. That’s how you serve a 405B model on 2 nodes (TP=8, PP=2).

Scaling Efficiency: How to Measure Honestly

Don’t trust marketing numbers. Run these yourself:

# Baseline: single GPU with the largest quant that fits
vllm bench serve --model llama-8b-fp16 --num-prompts 500 --request-rate 5
# TP=2 same request rate:
vllm bench serve --model llama-70b-fp16-tp2 --num-prompts 500 --request-rate 5

Metrics to record:

  • TTFT p50/p95/p99 (dominated by prefill and queue)

  • ITL p50/p95/p99 (inter-token latency — dominated by decode)

  • Throughput in output tokens/sec

  • Goodput (throughput within SLO; e.g., “tokens/sec where ITL_p99 < 100ms”)

  • GPU util per rank — if TP scales badly, you’ll see all-reduce waits on the Nsight timeline.

Report a scaling table. Format:

Config

Prefill throughput

Decode throughput

ITL p95

TTFT p95

Scaling

8B fp16 TP=1

X tok/s

Y tok/s

Z ms

W ms

1.0

70B fp16 TP=4

…

…

…

…

(measured/ideal)

70B fp16 TP=8

…

…

…

…

…

Bottleneck Diagnosis Playbook

“My TP=8 isn’t 8x faster than TP=1” — well, no, and here’s how to figure out why:

  1. Roofline the block. For your model, batch, seq: compute FLOPs per block, bytes moved per block. Compute FLOPs available and bandwidth available across N GPUs. Predict per-block time. If measured >> predicted → check step 2.

  2. Nsight Systems, look at NCCL rows. Are they solid and short (good) or long and gap-filled (bad)? If NCCL time > compute time, you’re comm-bound → reduce TP degree or increase batch.

  3. Check kernel launch overhead. vLLM captures CUDA graphs for common decode shapes; if you’re running with --enforce-eager, latency is 2-5x worse. Never benchmark eager unless debugging.

  4. Check GPU-GPU topology. nvidia-smi topo -m should show NV12 (fully NVLinked) between all TP ranks. Anything less (SYS = through PCIe, NODE = same NUMA but no NVLink) → move ranks or accept the cost.

  5. PCIe fallback. If NVLink Bridge is dead or unpaired GPUs, NCCL silently falls back to PCIe P2P (or worse, staging through host memory). Your “TP=2 is only 1.3x” mystery lives here.

The AMDA-Overlap Trick (Advanced)

Recent vLLM/SGLang optimizations overlap the second half of the all-reduce with the beginning of the next GEMM. This is only ~10-15% win on serving but is the state of the art. Ampere had AsyncTP; Hopper has better via TMA async loads. You don’t need to implement this, but knowing it exists explains why bleeding-edge engine versions get faster without hardware changes.

CUTLASS has explicit async-tp GEMM kernels. Some vLLM builds enable them via VLLM_USE_ASYNC_TP=1 — test if your workload benefits.

Data Parallelism at Serving Time

DP for serving means running N independent replicas of the same model, load-balanced by a router. This is:

  • The default throughput scaling strategy for models that fit in one GPU.

  • Combined with TP: TP × DP = world size. Example: 8 GPUs, 70B model. Either TP=8 (1 replica, ~500 tok/s per user) or TP=4 × DP=2 (2 replicas, ~350 tok/s each, but 2 concurrent users at that rate → much better throughput). Choose based on SLO.

  • SGLang exposes this cleanly via --dp. vLLM’s approach is typically “run multiple engine instances behind Ray Serve / K8s.”

Standard decision tree for a fixed hardware budget:

  1. Fits in 1 GPU at your quant + KV budget? → DP-only, N replicas.

  2. Fits in TP=2? Prefer TP=2 × DP=(N/2) over TP=N single replica for throughput.

  3. Doesn’t fit even in one node? → TP-max within node, PP across nodes.

The Zoho Angle

Your on-prem CRM customers get e.g. 4×L40S (48 GB each) or 8×H100. The napkin math you must produce in an hour:

Customer: “We have 8×H100 SXM, want to serve a 70B chat model to ~200 seats.”

Your derivation:

  • 70B in fp8 KV-fp8: 70 GB weights + KV that grows. Fits in 1 H100 for very small context, but not with realistic 8k context × concurrent users.

  • TP=2 with FP8: weights 35 GB/rank, ~45 GB KV budget/rank → ~200k tokens of KV total. At 8k avg per user → ~25 concurrent decode slots per replica.

  • 8 GPUs / TP=2 = 4 replicas. Total ~100 concurrent decode slots. At 5-8s active session per user × 200 seats × business-day duty cycle → fine, with prefix caching handling the system-prompt load.

  • Fallback: TP=4 for lower latency if p99 ITL SLO is strict → 2 replicas × ~50 slots = ~100 slots, similar throughput but ~30% lower per-user tok/s.

This is exactly the exercise you’ll do 50 times in your career if this becomes your specialty. Write the template once, tune it forever.

Exercises

  1. The scaling table. On rented 8×H100, serve Llama-3-70B in fp8 with TP={1,2,4,8}. Measure the table above. Explain each row.

  2. TP vs DP tradeoff. Same 8 GPUs. Compare TP=8/DP=1 vs TP=4/DP=2 vs TP=2/DP=4 for 100 concurrent users, ShareGPT trace. Which config maximizes goodput at ITL_p95<150ms?

  3. PP disaster. Run TP=1/PP=8 vs TP=8/PP=1 for a 70B. Confirm PP-only is a latency disaster. Measure how bad.

  4. Cross-node TP disaster. If you have 2 nodes with 4 H100s each and IB, try TP=8 (cross-node). Compare to TP=4/PP=2. Feel the pain that motivates the “never cross-node TP” rule.