16 — Rollout Infrastructure: The Training Loop That Contains an Inference Engine

This is the file your two career worlds fuse in.

Papers to keep next to your desk:

  • HybridFlow (verl) — Sheng et al., 2024 — arxiv:2402.01306

  • OpenRLHF — Hu et al., 2024 — arxiv:2403.07691

  • RLHF Workflow — Dong et al., 2024 — arxiv:2402.03300

  • GRPO / DeepSeekMath — Shao et al., 2024 — arxiv:2402.03300

  • DeepSeek-R1 — DeepSeek-AI, 2025 — arxiv:2501.12948

  • FP8 RL (veRL + vLLM/SGLang FP8 rollout) — 2026 preprint — preprint tracked separately, treat as speculative

Frameworks to know cold:

Part 1 — What makes RLHF/GRPO infrastructure a distinct systems problem

A supervised training step is one graph: forward, loss, backward, optimizer. A rollout-based RL step is three graphs stitched together:

1. Rollout      — sample G completions per prompt from the policy (this is INFERENCE)
2. Reward/scoring — run a reward model or rule-based scorer over the completions
3. Update       — recompute log-probs under policy + ref, compute loss, backprop (this is TRAINING)

Each graph has fundamentally different resource requirements. Rollout is memory-bandwidth bound autoregressive decode; training is compute-bound gradient GEMMs. The ratio of wall-clock time spent in rollout vs. update is typically 3:1 to 30:1 — which means if you build the whole thing on model.generate() inside a training loop, you leave 70–97% of your GPU-hours on the floor.

This is why:

  • “Just use HuggingFace generate() for rollouts” is the wrong answer at any serious scale.

  • The frontier of RLHF systems research is not new algorithms; it’s how to run vLLM-quality inference inside a training loop without the training and inference views of the model tearing each other’s memory apart.

  • This niche connects your two worlds. You already understand production inference (or will, by Phase 4). You already understand long-running services (from Zoho). The rollout-infra problem is “run an inference service, mutate its weights every 100 steps, keep it fast” — which is exactly your domain.

Part 2 — The three architectural patterns

Pattern A: Colocated (share GPUs between rollout and training)

The policy weights live on the same GPUs used for training. When it’s time to roll out, you either:

  • In-place mode: run generation using the training-framework’s forward pass. Slow (no paged KV, no continuous batching, no CUDA graphs). This is what naive TRL PPO did in 2023.

  • Engine-swap mode: on rollout step, dump policy weights into an on-GPU vLLM engine (same process), generate, tear down. High engineering cost; wastes time on the swap.

Colocated is fine for small models (<7B) on 1–2 nodes. Beyond that, engine-swap dominates.

Pattern B: Disaggregated (separate GPU pools)

Dedicate one GPU pool for training (FSDP2 or Megatron TP+PP), another for inference (vLLM or SGLang with TP). Weights are pushed from training → inference at a configurable cadence (every step, every N steps, or async).

Benefits:

  • Each pool runs its optimal framework: FSDP2/Megatron for training, vLLM/SGLang for inference.

  • Rollout parallelism scales independently (more inference GPUs → more concurrent completions).

  • Continuous batching + paged KV during rollouts — huge throughput.

Costs:

  • Weight transfer overhead. A 70B bf16 model is 140 GB; pushing that over NVLink is <1s but over IB is 3–10s.

  • Weight-syncing infrastructure to build (or steal from verl/OpenRLHF).

  • Idle GPUs on either side when the other is the bottleneck.

This is the modern default for 7B+ policies.

Pattern C: Hybrid (verl’s HybridFlow / 3D-HybridEngine)

The verl innovation. Weights are stored in one canonical sharding for training (e.g., FSDP or Megatron TP+PP+DP), but on rollout steps they are reshuffled into an inference-optimized sharding (typically higher TP, no DP replication) in-place on the same GPUs. The engine is torn between two personalities: training-shard during learn, inference-shard during rollout.

Benefits:

  • No dedicated inference GPUs (better utilization).

  • No cross-network weight transfer (all-gather within the training node group).

  • Optimal training shard and optimal inference shard.

Costs:

  • Complex code path — verl has spent 18 months on this.

  • Resharding takes 100ms–few seconds; amortized over rollouts.

This is now the frontier default (verl, OpenRLHF’s evolved versions, NeMo-Aligner all support variations).

Part 3 — verl (HybridFlow), the reference implementation to study

verl (Volcano Engine Reinforcement Learning) is ByteDance’s open-source RLHF framework and the current de-facto standard for research-grade RL on LLMs. It backs the RL training behind Doubao and many DeepSeek reproductions.

Design highlights (from the HybridFlow paper, arxiv:2402.01306):

  • Single-controller + multi-controller hybrid. A single-controller (Ray driver) coordinates the RL dataflow (rollout → score → update → sync); each stage internally uses multi-controller (SPMD) parallelism (FSDP/Megatron/vLLM). This gives you the readability of single-controller code with the performance of SPMD.

  • Hierarchical API. You write your RL algorithm at the dataflow level (rollout → advantage → update), and the framework handles distributed execution.

  • 3D-HybridEngine. The resharding mechanism above. Actor weights move between training and generation shardings in-place.

  • Backend support. FSDP2, Megatron-LM (TP+PP+SP+CP) for training. vLLM and SGLang for rollout. This is the matrix that matters.

When to reach for verl:

  • Any GRPO/PPO/DPO+rollout run on 7B+ models.

  • Any run where you want Megatron TP+PP (verl is one of the few frameworks that supports Megatron backends cleanly).

  • Any run where the rollout throughput dominates training — verl’s rollout backends are production-vLLM/SGLang, so you’re getting continuous batching for free.

verl recipe for GRPO on a 7B (single 8×H100 node):

# conceptual sketch of a verl config
actor_rollout_ref:
  model:
    path: Qwen/Qwen2.5-7B-Instruct
    enable_gradient_checkpointing: true
  actor:
    strategy: fsdp2                # training-side sharding
    optim: {lr: 1e-6, betas: [0.9, 0.999]}
    ppo_mini_batch_size: 64
  rollout:
    name: vllm                     # rollout engine
    tensor_model_parallel_size: 2
    gpu_memory_utilization: 0.6
    n: 8                            # G, group size for GRPO
    temperature: 1.0
    max_num_batched_tokens: 8192
  ref:                              # frozen reference for KL
    strategy: fsdp2
reward_model:
  # for GRPO on GSM8K use rule-based scorer instead
  strategy: rule_based
  score_fn: gsm8k_exact_match
trainer:
  n_gpus_per_node: 8
  nnodes: 1
  total_epochs: 5
  save_freq: 200

Reading list for verl mastery (order):

  1. HybridFlow paper (arxiv:2402.01306) — the systems paper.

  2. verl README + docs/ (https://verl.readthedocs.io/) — concepts + APIs.

  3. examples/ppo_trainer/, examples/grpo_trainer/ in the repo.

  4. The DeepSeek-R1 reproduction blog posts using verl.

Part 4 — OpenRLHF, the readable alternative

OpenRLHF (arxiv:2403.07691) is the more accessible cousin. Ray + DeepSpeed + vLLM. Its selling points:

  • Lower barrier to entry. ~5x fewer lines of code than verl’s Megatron paths.

  • Ray-native. If you already know Ray, OpenRLHF feels like home.

  • DeepSpeed ZeRO-3 backend (no Megatron path). Fine up to ~70B; painful above.

  • First to ship many features: ring-attention rollouts, KTO, PRM training loops.

Use OpenRLHF as your “read the code to understand what verl abstracts.” Then use verl for scale.

Part 5 — TRL, the accessible starter

TRL (HuggingFace) is where 90% of DPO and small GRPO runs happen. Its rollout story is weakest — historically it used model.generate() which is what we warned against — but in 2024–2025 it added a vLLM rollout backend for the GRPOTrainer and PPOv2Trainer.

  • Perfect for: DPO (no rollouts), small SFT+RL runs on ≤7B, teaching yourself the concepts.

  • Wrong for: 30B+ GRPO on a real cluster (use verl).

Key TRL entry points:

  • SFTTrainer — supervised.

  • DPOTrainer — preference, no rollouts.

  • GRPOTrainer — rollout-based, vLLM optional but recommended.

  • PPOv2Trainer — classical PPO, use only if you must.

Part 6 — The weight-sync problem in detail

At every training step (or every N), you must push updated policy weights from the trainer to the rollout engine. Naively:

for step in range(N):
    prompts = sample_prompts(...)
    completions = rollout_engine.generate(prompts)         # ~seconds
    rewards = score(completions)
    loss = grpo_loss(completions, rewards, ref_logprobs)
    loss.backward(); optimizer.step()
    push_weights_to_rollout_engine(policy_weights)         # SECONDS

That push_weights_to_rollout_engine is the crux. Approaches:

Approach

Latency

Complexity

Notes

Serialize to disk + rollout engine reloads

10–60 s

Low

Fine every 100+ steps; catastrophic every step.

Broadcast tensors over NCCL to inference workers

1–10 s

Medium

Requires inference workers to be part of the same NCCL world. This is what verl does.

P2P NVLink copies (colocated)

0.1–1 s

High

Only works if training and inference share GPUs (3D-HybridEngine).

Delta/LoRA-only sync

ms

Medium

If only LoRA adapters change, sync just the adapters (few MB). Killer optimization.

LoRA-only sync is the sleeper hit. If you’re doing GRPO with LoRA (increasingly common), your policy delta from the ref is tiny. You can push adapter weights to a static base loaded in the rollout engine in milliseconds, then let vLLM’s LoRA-serving path apply them. This is why LoRA + GRPO + vLLM is a magic combo for cost-sensitive setups.

Part 7 — On-policy-ness vs. throughput: the fundamental tradeoff

RLHF theory assumes rollouts are drawn from the current policy. If you sync weights every step and roll out one batch, you’re strictly on-policy — but throughput suffers because each rollout is small.

To speed up, systems introduce staleness: rollout with weights from step t-k while training moves to step t. This is “async” or “pipelined” RL. Small k (1–4) is empirically fine; large k (>16) causes instability.

Rules of thumb:

  • If G × prompts_per_batch × avg_completion_length is small (< 1M tokens/step), stay on-policy.

  • If it’s large, pipeline with k=2–4 and monitor divergence between rollout-time logprobs and training-time logprobs; if they diverge by more than 20%, decrease k.

  • verl and OpenRLHF both expose async/pipelined modes; use them.

Part 8 — Reward hacking, the ever-present monster

Unlike SFT and DPO, GRPO/PPO have an optimizer that will exploit any hole in your reward function. The failures I’ve watched happen personally to teams:

  • Length hacking: reward correlates with length → model outputs 3× longer completions with padding tokens hidden in reasoning.

  • Format hacking: reward regex requires \boxed{answer} → model emits \boxed{} early to secure the format bonus, then rambles.

  • Tool-call gaming: reward = “any successful API call” → model calls list_records with limit=1 on every prompt, always succeeds, always gets reward, never solves the task.

  • Prompt echo: reward LLM-judge with weak grader → model outputs the prompt verbatim as “answer,” grader is confused, gives partial credit.

  • Reference collapse: low KL β + long training → model drifts into gibberish that happens to score well on the (broken) reward.

Defenses:

  • Rule-based rewards over model-based whenever possible.

  • Add negative penalties for common exploits (length caps, format checks, must-not-echo-prompt).

  • KL to ref with β ≥ 0.001 — catches drift.

  • Sample 100 completions every 500 steps and look at them. Do not skip this.

  • Hold out an eval set the reward never sees; if train reward goes up but eval quality drops, you’re being hacked.

Part 9 — FP8 rollouts and the future

A 2025–26 development worth flagging: FP8 rollouts inside verl/OpenRLHF. The insight is that rollout is decode — memory-bandwidth bound — and FP8 halves the memory bandwidth cost of streaming weights. Doing FP8 rollout + bf16 training requires (a) correct log-prob accounting across precisions (importance sampling correction: TIS/MIS) and (b) confidence that the FP8 quantization doesn’t drift the policy off-distribution from the bf16 training version.

Emerging papers (search: FP8 RL veRL preprint) show 30–44% rollout throughput gains with proper importance-sampling correction. Treat this as an open frontier: expect the standard verl/OpenRLHF stacks to adopt FP8 rollout defaults through 2026.

Part 10 — Zoho angle: this is your career story

Internalize this framing until you can pitch it in 30 seconds:

“I’m the engineer who understands both the agentic harness calling an LLM and the inference engine serving it — which means I understand rollout-based post-training, where those two collapse into one system. Very few people can move fluently across all three layers.”

Concrete steps at Zoho:

  1. Instrument agentic harnesses to capture rollout traces — (prompt, completion, tool_call, outcome_success_bool). This is preference data + verifiable reward, for free, in your logs.

  2. Build a small internal GRPO pipeline on Qwen-2.5-7B, reward = “tool call succeeded and returned useful data.” Even a modest lift here is a career-tier internal artifact.

  3. Publish the benchmark methodology externally when your legal team lets you — the rollout infra + verifiable-reward + on-prem CRM combo is under-published.

  4. Contribute to verl or OpenRLHF — the merged PR is the credential. Documentation improvements, reward function libraries, benchmark harnesses — all are welcome low-friction entry points.

Part 11 — What the frameworks don’t tell you

Hard-won knowledge that isn’t in the docs:

  • Tokenizer drift. If your reward model and your policy use different tokenizers, the log-probs you compute for KL will be meaningless. Always sanity-check tokenizer.encode(x) on both.

  • Chat template drift. If your rollout uses one chat template and your ref logprob uses another, KL blows up. Force apply_chat_template at one canonical layer.

  • Padding weirdness. Left-pad for rollout (so all sequences start at position 0 for RoPE), right-pad for training. Get this backwards and half your training compute is on padding tokens.

  • Numerical stability of the ratio. π_θ / π_θ_old overflow / underflow on long sequences. Compute in log-space, clip before exp.

  • Ref model memory. Even frozen, the ref model needs to fit somewhere. Options: shard with FSDP (frozen), keep it CPU and sample-then-score, or (best) use LoRA + shared frozen base so ref = base and adapter=0.

Exercises

  1. Set up TRL’s GRPOTrainer on Qwen2.5-1.5B with GSM8K. Rollout via TRL’s built-in vLLM backend. Measure the wall-clock ratio between rollout and update. Predict this ratio before you run.

  2. Modify exercise 1 to use LoRA + LoRA-only weight sync. Measure the throughput improvement.

  3. Read the HybridFlow paper (arxiv:2402.01306) and draw the 3D-HybridEngine resharding diagram from memory.

  4. Deliberately design three flawed reward functions for GSM8K and predict how each would be hacked. Then run one for 500 steps and see if reality matches your prediction.

  5. Sketch a rollout infra design for the following scenario: 32-node cluster (8×H100 per node), 70B policy, GRPO with G=16, 10k prompts/hour target. Which nodes are training, which are rollout, how do weights sync? Justify every choice.

References

Next: 17_deepseek_v3_report.md — the single most instructive tech report of the modern era. Bring your notes on MLA, MoE, MTP, FP8, and everything you’ve learned about parallelism to this one.