09 — Disaggregated Prefill/Decode Serving

Why this file: In one sentence: prefill is compute-bound and decode is memory-bound, so putting them on the same GPU wastes half the silicon in both phases. The last two years of production LLM infrastructure have been about splitting them apart. Understanding this deeply — DistServe’s motivation, Mooncake’s KV-centric implementation, vLLM’s and SGLang’s evolving PD-disaggregation — is the single most important architectural literacy for staff-level inference engineering right now.

Reading (in this exact order):

  1. DistServe (Zhong et al., OSDI ‘24) — arxiv:<phone_number_or_numberic_id_or_random_id_113>. The paper that named the problem and framed “goodput” as the honest metric.

  2. Mooncake (Qin et al., FAST ‘25 Best Paper) — arxiv:<phone_number_or_numberic_id_or_random_id_114>. Kimi’s KVCache-centric architecture. Best-paper claim verified: https://www.usenix.org/conference/fast25/presentation/qin — “Awarded Best Paper!”

  3. Splitwise (Patel et al., ISCA ‘24) — Microsoft’s earlier related work; useful for the phase-characterization framing.

  4. Mooncake open-source project: https://github.com/kvcache-ai/Mooncake

  5. NIXL / LMCache: the KV transfer libraries underneath modern disaggregated serving.

  6. Nexus (arxiv:<phone_number_or_numberic_id_or_random_id_115>) — 2025-2026 alternative: proactive intra-GPU PD disaggregation instead of cross-pool.

Part 1: Why Prefill and Decode Fight

The problem in one figure (imagine drawing it):

Request arrives:  ─┬─ Prefill (10k tokens) ─┬─ Decode (500 tokens) ─┐
                   │  Compute-bound          │  Memory-bound         │
                   │  ~1-2 sec on H100       │  ~10-15 sec, per-token│
                   │  Utilizes tensor cores  │  Streams weights+KV   │
                   ▼  Utilizes ~80% FLOPs    ▼  Utilizes ~15% FLOPs  ▼
                                             ▲
                              But other users' decodes have to wait
                              during your long prefill (head-of-line blocking).

Consequences of co-locating prefill and decode on the same GPU:

  1. TTFT vs ITL tradeoff is stuck. Batching more decodes with a prefill boosts throughput but wrecks ITL (the prefill hogs the SM). Batching fewer preserves ITL but starves throughput.

  2. Chunked prefill (Sarathi-Serve) was the first fix: split prefill into small chunks and co-schedule with decode. Great, but it tunes the knob rather than removing it.

  3. Different parallelism preferences. Prefill wants high-TP for latency (big GEMMs benefit from parallel compute). Decode wants moderate TP for capacity + high batch for throughput. One fixed TP degree can’t be optimal for both.

  4. Different SLOs. TTFT and ITL are decoupled from the user’s view; you should be able to scale them independently.

The solution: run prefill on GPU pool A, decode on GPU pool B, transfer KV cache between them.

Part 2: DistServe (OSDI ‘24) — The Framing

Paper: DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang (UCSD + PKU). arxiv:<phone_number_or_numberic_id_or_random_id_113>.

Core claims:

  • Prefill and decode have opposite resource profiles. Interleaving them on shared GPUs causes prefill-decode interference that hurts both TTFT and ITL.

  • Different phases benefit from different parallelism strategies. Prefill likes TP for latency; decode likes larger batch, sometimes lower TP.

  • “Goodput” = tokens/sec that meet BOTH TTFT and ITL SLOs. This is the honest metric. Papers reporting only throughput are hiding tail latency.

Result: DistServe can handle 7.4x more requests or 12.6x tighter SLO than an SLO-optimized colocated system, on the same hardware.

Architecture:

  • Prefill instances: high TP within-node, sized for TTFT SLO.

  • Decode instances: larger batch, potentially lower TP, sized for ITL SLO.

  • KV transfer: after prefill completes, KV cache moves from prefill GPU HBM → decode GPU HBM.

  • Scheduler co-optimizes the split (how many GPUs to each pool) with the parallelism per pool, subject to the cluster’s bisection bandwidth.

The transfer is not free. KV cache for 10k tokens of a 70B model is ~5-10 GB. At NVLink speeds (~450 GB/s) it’s ~20 ms; at IB NDR (~50 GB/s) it’s ~200 ms. Whether disaggregation wins depends on:

  • (prefill_time_saved_by_uninterrupted_prefill) + (decode_time_saved_by_uninterrupted_decode) > (KV_transfer_time)

  • For long outputs (>200 tokens), decode savings dominate → disaggregation is a huge win.

  • For very short outputs (<50 tokens), the KV transfer may not amortize → don’t disaggregate. Real systems route accordingly.

Part 3: Mooncake (FAST ‘25 Best Paper) — The Production System

Paper: Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot — Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu (Moonshot AI + Tsinghua). arxiv:<phone_number_or_numberic_id_or_random_id_114>. Best Paper Award at FAST ‘25 — confirmed via USENIX (https://www.usenix.org/conference/fast25/presentation/qin) and Kimi_Moonshot’s official announcement.

Deployed as: the serving platform for Kimi, Moonshot’s chatbot, at scale. This is not a research prototype — it processes real user traffic on A800 and H800 clusters.

The Key Insight: KVCache is the Center

Most systems think: “models are the resource, KV is a per-request thing.” Mooncake inverts this: KVCache is the shared, distributed resource; prefill and decode are consumers of it.

Architecture:

                     ┌───────────────────────┐
                     │  Conductor (scheduler)│  ← predicts, admits, routes
                     └──────┬────────────────┘
                            │
           ┌────────────────┼─────────────────┐
           ▼                ▼                 ▼
      Prefill Pool     Decode Pool       Distributed KVCache
      (Kimi-P nodes)   (Kimi-D nodes)    (CPU DRAM + SSD across cluster)
           │                │                 ▲
           └──── KV write ──┼─── KV read ─────┘
                            │
                     RDMA fabric (IB / RoCE)

What lives in the distributed KV cache?

  • Underutilized CPU DRAM on every node in the cluster (idle GPU nodes still have TB of DRAM sitting there).

  • SSD/NVMe tiers for colder cache entries.

  • Managed as a global block store. Prefixes (system prompts, few-shot examples, conversation histories) become cluster-wide cache hits, not just per-replica.

KV transfer via RDMA: GPU HBM → CPU DRAM → RDMA over IB → CPU DRAM → GPU HBM. GPU-direct RDMA is used where the topology allows. Fast enough that KV transfer overlaps with the tail of prefill and the head of decode.

The Prediction-Based Scheduler

Standard LLM serving lets requests join a queue and get scheduled when a slot is free. Mooncake predicts:

  • Expected output length (from a light predictor).

  • Expected KV cache size consumed.

  • Expected TTFT and ITL given current pool loads.

Using this, the Conductor does prediction-based early rejection: if serving this request would violate SLOs even under optimistic scheduling, reject it before it consumes prefill compute. This is a backpressure mechanism, essential at scale.

The Numbers

From the paper and USENIX abstract:

  • Up to 525% throughput increase in simulations (498% in the tweet’s number).

  • 115% more requests handled in real-world scenarios on A800 clusters, 107% on H800.

  • Best paper at FAST ‘25 — systems community recognition that KV-centric disaggregation is the right architecture for chatbot serving at scale.

The Open-Source Story

Mooncake was open-sourced: https://github.com/kvcache-ai/Mooncake. The Mooncake Transfer Engine is the KV transfer library, now a first-class citizen in vLLM and SGLang integration paths. Read the repo README for the block-manager and store-engine APIs.

Part 4: Splitwise (ISCA ‘24) — The Microsoft Precursor

Paper: Splitwise: Efficient Generative LLM Inference Using Phase Splitting — Patel et al. Microsoft Research. Ran alongside DistServe conceptually.

Key contribution: characterization study showing that prefill and decode have distinct compute/memory profiles across different generations of GPUs (A100 vs H100), and that heterogeneous fleets can specialize — e.g., prefill on H100, decode on A100/L40S. Cost-optimal.

Practical takeaway: if you have a heterogeneous fleet (mixed H100 + L40S), route prefill to the higher-FLOPs card and decode to the higher-$/GB-of-bandwidth card. This exact play is on Zoho’s enterprise horizon.

Part 5: vLLM PD-Disaggregation (Current State, 2025-2026)

vLLM has been adding PD-disaggregation over 2024-2025:

  • v0.6.x+: experimental disaggregated support via --kv-transfer-config.

  • v0.7+: more mature; integration with NIXL (NVIDIA’s KV transfer library) and LMCache (KV cache pooling library from UChicago Junchen Jiang’s group).

  • Two-side deployment: launch prefill-role instances (--kv-role kv_producer) and decode-role instances (--kv-role kv_consumer) with a shared config for KV transport.

Example sketch (subject to version drift — always check current docs):

# Prefill instance
vllm serve model --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer","kv_ip":"...", "kv_port":...}'

# Decode instance
vllm serve model --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer","kv_ip":"...", "kv_port":...}'

# Router in front distributes requests across both.

Check: https://docs.vllm.ai/en/latest/features/disagg_prefill.html for current syntax.

KV transfer backends integrated into vLLM:

  • MooncakeStore — the Kimi library.

  • NIXL (NVIDIA Inference Xfer Library) — GPUDirect RDMA-first, part of the NVIDIA Dynamo stack.

  • LMCache — provides broader KV cache pooling (also does non-prefix reuse via CacheBlend techniques).

Part 6: SGLang PD-Disaggregation (Current State)

SGLang added PD-disaggregation in 2025 as first-class:

  • python -m sglang.launch_server --disaggregation-mode prefill|decode

  • Uses Mooncake Transfer Engine or NIXL underneath.

  • Combines with SGLang’s RadixAttention prefix cache — the prefill pool hits the shared cache before doing GEMMs, so many prefills become sub-millisecond hits.

Check: https://docs.sglang.ai/references/disaggregation.html for current syntax.

Part 7: The Orchestration Layer Above

Engines (vLLM/SGLang) are the executors. The layer above wires them into a serving cluster:

System

Origin

What it does

NVIDIA Dynamo

NVIDIA (2024+)

K8s-native orchestrator for disaggregated LLM serving. Handles routing, autoscaling, KV transfer topology. Includes NIXL as the transfer primitive.

llm-d

Red Hat + Google + IBM (2024)

Open orchestrator built on K8s + vLLM/SGLang. KV-cache-aware routing.

LMCache

UChicago

KV cache pooling library. Also serves as a shared cache store across engine instances.

Mooncake Store

Moonshot AI

Distributed KV block store, open-source. Used by vLLM/SGLang integrations.

KServe / Ray Serve

Kubeflow / Anyscale

Generic model serving with LLM extensions. Older, still deployed.

AIBrix

ByteDance (2025)

Similar space — disaggregated inference at fleet scale.

These are your “Layer 6” from the base roadmap — orchestration/scheduling.

Part 8: When Disaggregation Pays (Decision Matrix)

Workload

Disaggregate?

Why

Chatbot with long histories (Kimi-like)

Yes, big win

KV reuse is enormous, prefill-decode ratio varies wildly per request

Agentic tool-calling loops (your Zoho case)

Yes

Repeated system prompts + long tool traces → massive KV reuse potential

Code completion (short prompts, short outputs)

Marginal

KV transfer overhead may not amortize

Long-form generation (short prompt, long output)

Yes

Decode dominates, prefill-decode interference matters

Batch offline scoring (short prompts, greedy 1-token)

No

Prefill is 99% of the work; just batch it.

Very small models (7B)

Usually not

Interference is small when a whole request fits in a fraction of one GPU

Frontier MoE (DeepSeek-R1 class)

Mandatory

Wide-EP + PD-disagg is the ONLY economical way to serve at scale

Part 9: The Zoho Career Angle (Emphasis)

Your agentic-harness day job at Zoho is the archetype of the workload disaggregation was invented for:

  • Multi-turn conversations with growing histories.

  • Tool-calling loops that resend the same context repeatedly.

  • On-prem CRM customers who need predictable p99 latency SLOs — exactly what “goodput” as a metric captures.

  • Heterogeneous fleets (mixed L40S + H100 boxes) where Splitwise-style specialization pays.

The single most credentialing artifact you could produce in 2025-2026-<phone_number_or_numberic_id_or_random_id_120>:

A public benchmark comparing colocated vs disaggregated serving of a real agentic workload (multi-turn tool-calling, ShareGPT-style trace + tool responses), with measured TTFT/ITL/goodput on both. Nobody has published a great one yet with the workload profile you have insider access to.

This is a paper-shaped artifact and a job offer artifact.

Part 10: Exercises

  1. Read the DistServe paper end-to-end. Extract:

    • The exact formula for placement-under-bandwidth constraint (§3.2).

    • The goodput definition (§2).

    • Why they use different TP degrees for prefill vs decode pools. Write a 1-page summary. This is a required artifact.

  2. Read the Mooncake paper. Extract:

    • The chunked KV cache block layout (they use fixed-size blocks like paged attention, but distributed).

    • The prediction model for the scheduler (§4).

    • The failure modes and how prediction-based early rejection handles them.

  3. Set up a toy PD-disaggregated vLLM. Two vLLM instances on one machine (or two nodes), one as producer, one as consumer, with NIXL or MooncakeStore between. Run 100 requests through it, compare TTFT+ITL to colocated.

  4. Napkin math the KV transfer cost. For Llama-3-70B fp8, KV bytes per token = 2·80·8·128·1 = 163 KB. For a 10k-token prefill, KV = 1.6 GB. At IB NDR 50 GB/s → 32 ms transfer. Compare to a 500-token decode at 20 ms/token = 10 sec. Transfer is 0.3% of decode time. Verdict: transfer cost is essentially free for realistic workloads.

  5. Route a request stream. Build a tiny router (~200 lines Python) that decides: this request → colocated vLLM (short prompt+output) vs disaggregated (long prompt or long output). Measure aggregate goodput vs a monolithic router.

Big Findings to Internalize

  1. Mooncake best paper at FAST ‘25 is the current north star for KV-centric disaggregated serving. It has changed how new engines are designed.

  2. KV cache is a distributed resource, not a per-request thing. Sharing it across the cluster (Mooncake Store, LMCache) unlocks the biggest single throughput gains available in 2025-2026.

  3. Goodput, not throughput. Any benchmark that doesn’t report tail latency + SLO adherence is marketing.

  4. Disaggregation isn’t universally better — it’s workload-dependent. Chunked prefill (Sarathi-Serve) remains competitive for many workloads. Both should be in your toolkit.

  5. NVIDIA Dynamo + NIXL is the vendor-blessed stack; Mooncake + vLLM/SGLang is the open-community stack. Both use the same abstractions. Learn both.