08 — CUDA Graphs in vLLM

Why decode captures CUDA graphs, why prefill doesn’t, and why “eager mode” is usually slower.

The problem: kernel launch overhead at small batch

A single decode step for a small batch (say 4 sequences) on an 8B model runs ~250-400 kernel launches: attention, projections, RMSNorm, SwiGLU, elementwise adds, sampling. Each kernel launch on CUDA takes roughly 5-15 μs of CPU-side overhead before the GPU starts work — setting up the launch args, checking the driver’s command queue, etc.

Do the math:

  • 350 launches × 10 μs = 3.5 ms of CPU overhead per decode step

  • On H100, the actual GPU work for that decode is ~2-4 ms

CPU overhead is comparable to GPU work. The GPU sits idle waiting for the CPU to issue the next kernel. This is called launch-bound and it’s the dominant bottleneck for small-batch decode.

(At large batch the GPU work grows but launch count stays fixed, so launches become negligible. Chunked prefill also isn’t launch-bound because each kernel is large. Small-batch decode is the specific regime that hurts.)

The solution: CUDA graphs

CUDA graphs are a mechanism to record a sequence of kernel launches once, then replay the entire graph with one API call (cudaGraphLaunch) later. The recording captures the entire structure: kernel calls, memory operations, dependencies. Replay pays ~5-10 μs total, not 5-10 μs per kernel.

So 3.5 ms of CPU overhead collapses to ~10 μs. ~300× reduction in CPU-side latency, translating to a 20-50% end-to-end speedup on small-batch decode.

What vLLM V1 does

vLLM captures one CUDA graph per batch size (padded to a set of discrete sizes like 1, 2, 4, 8, 16, 32, 64, 128, …, max_num_seqs).

On startup:

  1. For each captured batch size B:

    • Allocate dummy input tensors sized for B.

    • Run one forward pass under torch.cuda.CUDAGraph() capture mode.

    • Store the graph handle keyed by B.

At runtime, per decode step:

  1. Round the actual batch size up to the nearest captured size B’.

  2. Copy inputs into the captured input buffers.

  3. Launch the graph for B’. One CPU-side call.

  4. Read outputs from the captured output buffers.

The cost: extra GPU memory for the input/output buffers of each captured size (few hundred MB total) and a few seconds of startup time to capture the graphs.

Why prefill doesn’t get graphs

Prefill inputs have dynamic shape: prompt lengths differ across requests, and chunked prefill adds another axis of variation. Each shape combination would need its own graph, and the combinatorial explosion (batch_size × prefill_length × chunk_size) makes capture impractical.

Also: prefill kernels are large; launch overhead is small compared to GPU work. Prefill isn’t launch-bound. No point capturing what isn’t slow.

vLLM V1 runs prefill in eager mode (regular Python-driven kernel launches) and decode from a captured graph. --enforce-eager disables graph capture entirely (useful for debugging, always slower in production).

Interaction with other features

With chunked prefill

Hybrid steps (some decodes + some prefill chunks) can’t use pure decode graphs. vLLM V1 handles this by either:

  1. Running the whole hybrid step in eager mode, or

  2. Capturing a limited set of hybrid graphs for common shapes.

Details vary by vLLM version; current V1 uses a mix. Trace it in vllm/v1/worker/gpu_model_runner.py around the capture logic.

With speculative decoding

Spec decoding also uses graphs (the drafter and the verify step are both captured). Total captured graphs increase, memory cost grows.

With torch.compile

vLLM V1 leverages torch.compile heavily (Inductor-generated fused kernels replace hand-written ones for many ops). The compiled Inductor kernels play nicely inside CUDA graph capture — they’re just kernels like any other. The two mechanisms compose.

When eager mode is actually faster

Rare, but real cases:

  1. Development / debugging. Graphs hide the failure site — the whole graph replay fails as one, error messages are worse. Use --enforce-eager when hunting a bug.

  2. Highly variable batch sizes. If your workload constantly straddles uncaptured sizes, the padding waste can offset graph benefit. Usually not the case in production (batch sizes cluster).

  3. Extremely small models on fast GPUs. For a <1B model on H100, GPU work is so cheap that even the padded batch size wastes noticeable compute. But you shouldn’t be running a <1B on H100 anyway.

  4. Certain custom ops that don’t capture cleanly. Rare; XGrammar masking used to have this issue, largely fixed.

For 99% of production configs, CUDA graphs on = free 20-50% speedup. Default in V1; leave it.

The debugging story

Common symptom: enabling graphs causes a rare crash you can’t reproduce with --enforce-eager. Usual culprits:

  1. Dynamic control flow inside the model. Any Python-level branching on tensor values breaks capture. Some models have this in edge cases.

  2. CPU-GPU sync inside a captured kernel. .item(), .cpu(), .numpy() on a captured tensor breaks the graph.

  3. Non-deterministic ops whose plan differs across launches (some cuBLAS heuristics).

Fix path: --enforce-eager to isolate, then narrow to specific model op with graph capture disabled per module, then fix (usually rewrite the branch as a mask).

The measurement to do once

Benchmark the same model on the same hardware with:

# eager
vllm serve MODEL --enforce-eager
# then bench serve

# graph (default)
vllm serve MODEL
# then bench serve

Compare TPOT p50 at batch size 4 (small) and 64 (large). Expected:

Batch size

Eager TPOT

Graph TPOT

Speedup from graphs

1

~30 ms

~20 ms

1.5×

4

~35 ms

~24 ms

1.5×

32

~55 ms

~48 ms

~1.15×

128

~120 ms

~115 ms

~1.05×

(Illustrative; run yours.) The speedup shrinks as batch grows, because GPU work grows to dominate. This is the launch-bound pattern.

Reading exercise

  1. Open vllm/v1/worker/gpu_model_runner.py. Find the graph capture logic (search for cuda_graph or CUDAGraph).

  2. Answer: how are the discrete captured batch sizes chosen? Is it a power-of-two ladder, an arithmetic progression, or config-driven?

  3. Answer: what happens if a request comes in with a batch size larger than the max captured? (Hint: look for a fallback path.)

The intuition to internalize

At small batch, the CPU is the bottleneck, not the GPU. CUDA graphs fix this by paying the CPU cost once at capture time and amortizing it forever. Every serious inference engine now captures graphs for its steady-state decode path. Every serious engineer knows why.

When someone asks “why is decode faster in vLLM than in eager PyTorch?” — the top three answers are: paged KV, custom attention kernels, and CUDA graphs. This file was the third one.